> Anyone that has worked on large, complex system knows that the margin of error in uptime and downtime is often whether the right person is within arms’ reach of their laptop.
Is this true?
Shouldn't giant tech companies obsess about reducing the need for human intervention?
Giant tech companies do obsess about reducing the need for human intervention. Teams in my org at AWS kept track of failures/intervention rates per thousand instances. If it gets too high, it means you're spending too much engineering effort resolving on-call issues and need to fix it.
PagerDuty is a multi billion dollar company for a reason, and they're not even the only company doing what they do.
I don't think it's relevant if one has worked in a giant company to understand how bad on call can be, every engineer knows that. I personally assume on-call is much worse/harder/nerve wracking in bigger companies
They do. You automate recovery for all the failure modes your system has encountered. Then the system promptly fails in a new way you've never seen before.
Often because some totally different part of the system fails when scaling to new levels.
When I was oncall for my team, I found there were less pages, less issues, and the system was generally more stable.
Entropy, leading to availability problems, grows with rate of production changes.
If no one touches the code, my guess is the system is more stable rather than less.
Of course the solution can't really be "let's not deploy anything, just to be safe", because then your competitors are going to launch new features and leave your product behind.
For example, one day SRE got alerted that a bunch of expensive accelerators were unexpectedly shutting down and not restarting production. SRE has to reach out in this case to the SWEs who build/designed the system to ask some clarifying questions. Together, the SREs and SWEs form a series of hypotheses about the cause, ultimately discovering an entirely unanticipated failure mode.
I think I'm one of the few people in the world who has attached a $100K oscilloscope to the voltage regulator on a machine learning accelerator to debug why a specific training job that did a series of convolutions at a highly specific rate would cause a DC-DC regulator to act like an AC source. It took far, far longer to write and deploy the rule that detected this problem in prod than it took us to identify the problem and stop the killer job.
The general philosophy at these orgs is that the same failures should never happen _again_. So you build automation and safeguards protecting the system from the failure modes you know.
However, any complex system will have failure modes you don't know. There might be new software, new features, new APIs etc. going out that interact in complex ways. So complex systems will fail in very interesting ways. So the general philosophy in operating these systems is:
1. First get the system back into a state in which the problem is mitigated. 2. Apply some short term hacks, rollback any suspicious recent changes. 3. Have someone go a bit deeper and try to root cause what caused the failure, have a discussion about it with impacted teams (often called a postmortem) and come up with long term fixes that reduces or eliminates the root cause from happening again.
The companies where it was not true, was a pleasure to work in. The management, from C-suits downwards knew what they were doing. In the others, it was a total chaos.
Of course, if everything is fixed with duct tape, you need firemen ready to act. If everything is solid and robust, there can be small outages, but nothing too critical.