Hacker News Reader

Even a staged rollout over a few hours would have made a huge difference here. "Slow" in the context of a rollout can still be pretty fast.

But it can also still be way too slow in the context of an exploit that is being abused globally.

Sure but GP is praising "deploy so slowly that people complain."

Seriously like rolling out on some exponential scale even over the course of 10 minutes would have stopped this dead in its tracks

In CrowdStrikes case, they could have rolled out to even 1 million endpoints first and done an automated sanity/wellness check before unleashing the content update on everyone.

In the past when I have designed update mechanisms I’ve included basic failsafes such as automated checking for a % failed updates over a sliding 24-hour window and stopping any more if there’s too many failures.

They need a lab full of canaries.

yeah, I don't get the "we couldn't have tested it" crap, because "something happens to the payload after we tested it". Create a fake downstream company and put a bunch of machines in it. That's your final test before releasing to the rest of the world.