"You can't seriously tell me that the unhappy leg of the code path has no test coverage."
Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.
Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.
Cloudflare outage on December 5, 2025 [1]:
> However, we have never before applied a killswitch to a rule with an action of “execute”.
Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO.
They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went.
A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else).
A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before.
Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse.
The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated).
The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification.
I've seen cases which had:
1. Retries on the client side (browser or mobile app). 2. Retries on the BFF. 3. Retries on Microservice A used by the BFF. 4. Retries on Microservice B used by Microservice A. 5. Retries on Critical Service C used by Microservice B.
Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C.
We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day.
A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination.
[1] This was the CloudFlare issue mentioned in this thread https://blog.cloudflare.com/5-december-2025-outage/
As said GP, yes this is exactly what I meant and "happy-path" was an unclear choice of words. They optimize for the unhappy path on the good days, and make the bad days much worse.
1. People add retries w/o backoff. Now we have a retry storm.
2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure.
3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns.
4. People find ways to fetch from multiple data sources to make a composite object but don't cache/reuse data they fetched, so one data source being down causes DOS on _other_ data sources because of retries.
5. People add failover mechanisms where all failures happily pick the same failover target because, ya know, it has the most free capacity.
6. People underestimate connection setup cost so "failover" causes huge increase in overall load due to connection setup (often tls setup) causing huge influx of "new" cpu work in a loaded system. CPU spike causes unhealthy destinations, causing more failovers, causing metastable failure.
7. People realize purely local decisions aren't optimal so add a layer of global health metrics (e.g. global retry budgets) but these systems add a time delay to the metric (ie its the view of the world 5ms ago). This delay makes their naive control algorithm go into oscillations or divergence spraying stuff everywhere.
All of these things have a similar pattern that when things are almost all good, they will make the system more robust. You'll get an extra 9 of reliability on good days. But they make everything far worse when things are bad.
It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.
I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.
Ergonomics are highly variable, so it might not help someone else, but for me it did the opposite of "give me carpal tunnel syndrome."
Just pick a specific task that you do routinely (say markdown editing) and just use neovim for that one task - eat the elephant one bite at a time and markdown is generally small enough/light enough that you won't get out into the weeds trying to make neovim the perfect IDE in the process.
I recently tried zed but it ddos’d my nas over an smb share lol
Intellij for basically all software development that is more than 2 minutes.
They can afford to do this exhaustively because the test criterion is universal: they are testing ACIDity - the database is either in the new state or the old state. Never in between or corrupted.
Any other kind of system wouldn't be so easy to test for malloc failure, since you couldn't check for things like a successful response.
At a non-tech-company, that’s not the case. Which means you end up with something much closer to the movie “Office Space”, where software is just considered a necessary evil that’s worked on by what’s essentially the software development arm of an IT department, where any opportunity to cut costs or headcount is considered a good idea, where “best practices” take a back seat to “any practice that Bob can make work.”
A lot of companies these days are somewhere between the two: their main product is not software, but the company depends on a website to interact with a large number of customers: banks, insurance companies, web stores, etc. In that case, you tend to get the worst of both worlds, naturally.
It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?
That means nothing, especially with all the layoffs.
One thing that I learned over the years is never mystify "code quality". Maybe you can say a certain team/project produces high quality code, and maybe Google's overall quality is better than my company's (considering their bar of hiring), but you want to avoid generalizing that, especially at scale.
The average software engineer at a large company ships just enough code at good enough quality to earn a salary, nothing fancy about it.
MS has layers, there are A-teams, B-teams, and so-on. Simultaneously they are dropping both world class work and, to your point, a statistically much larger pool of just pretty good code.
A lot of that work is perfunctory and somewhat bloated/weakened by the scale of publishing and their internal tech-political requirements. Components for component I’ve seen in-house replacements for many MS projects from mid-tier devs that are better, smaller, and less brittle over time.
I’d compare it to baking: a home cook or tiny bakery have the advantages of time and focus, perpetual consistency at scale is a whole different baking challenge, one that doesn’t necessarily yield better individual loaves. It doesn’t mean WonderBread is bad at bread, just that they’re optimized for something other than maximal loaf quality.
Microsoft can afford to do a lot of things, but why when you can squeeze a bit more profit out.
It is miraculous how they continue to fail upwards somehow. I suppose it's because they just vampirically sustain themselves by acquiring other companies and sucking them dry until they are a husk of what they once were -- like Github, Activision/Blizzard, LinkedIn, Skype, etc.. I remember when those companies used to actually produce good products and services.
Same with Oracle. Both of them are good at getting contracts and then executing them not poorly enough that it's breach of contract but not well enough that the customer stops paying. That's their business, not software.
Would you want to go and spend billions to migrate off Microsoft/Oracle when you could just not and get to the next quarter.