Hacker News Reader

Comment by wrs | Hacker News Reader

When deciding whether to do an incident investigation for a near miss, one aspect to consider is whether it was caught by a safety system as designed, or caught by a lucky accident. The latter should be top of the priority list.

E.g., Bad package deployed to production. Stopped because it didn’t have the “CI tested” flag: low pri. Stopped because someone happened to notice a high CPU alert before the load balancer switched over: high pri.

View on HN

It can be a good idea to count and record these safety net activations for reporting purposes. If you do so, the discussion of e.g. "Why do we run replicas for our databases" becomes a very simple discussion: It has simplified us maintenance X times this year (by avoiding downtime to to switch overs), and it has avoided Y outages (again due to automated switch overs) - recovery of the databases after the outage took Z hours, which would have been downtime. Suddenly you've moved the vague "We need it for safety and availability reasons" into concrete admin-hours and downtime-minutes.

Or, "Can't we skip all these tests/other long-running quality process?"

Definitely, counting the ones you have is great data for justifying their existence, and can help identify places you should perhaps put a check farther upstream.

> Stopped because it didn’t have the “CI tested” flag: low pri.

Agree, certainly lower priority than the "caught by chance" case, but I wouldn't say "low priority". That system is the last line of defence and the fact that something is hitting it is definitely a large concern.

In the CrowdStrike's fiasco last year, their last line of defence, "Content Validator" failed.

"On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data."

https://www.crowdstrike.com/falcon-content-update-remediatio...

Oh absolutely, if you have the budget then do them all. I just meant “lower pri” than the other.

My assumption is in most orgs you’ll find it hard to take time to investigate any of them, so you have to place your bets.

Reply on news.ycombinator.com