Light on technical and light on details.
Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.
Putting the actual blast radius aside, this whole thing seems a bit amateurish for a "security company" that pulls the contracts they do.
- They don't do enough or the right kind of smoke tests.
- They don't do exponential-canary deployments with an ability to rollback, and instead just YOLO it.
- They don't appear to have a customer-side security / client platform team update approval gating change control process for software updates or for definitions (or whatever they use).
This is fundamentally laziness and/or incompetency.
From management - that couldn't care less about low visibility, low impact projects
The company that lobbied the hardest and paid the most in bribes got the contracts.
You notice how this outage affected hospitals and airlines? There is a strong tendency in software sales for industries to align around one or two leaders. Oh, American chose Crowdstrike? Maybe we at Delta should just do what they did. Or literally Delta hires the VP from American to be their CISO and he just does what he did before.
Vendor selection is hard and buyer's remorse is frequently hard to deal with once you've sunk cost into a migration.
This seems very amateurish for companies who regularly talk professionally to win said contracts , whether the best product or not.
My guess is C-suite, crisis consultants and lawyers are involved heavily so the actual engineering folks have little voice now in any communication and we get stuff like this.