I find that astonishing. Outside Silicon Valley, I think that it would be quite unusual to leave $50 billion of plant running overnight, but pinch pennies by not paying anyone to stay up and check the oil levels during the dog watches.
> In a follow-up email two days later, Pham addressed criticism of his decision to email the staff about the engineer’s mistake. "It came to my attention that some people in the org think that my note to the company is overly critical and has the feel of throwing people under the bus,” he wrote.
> “I don't want to create a culture [where] people are fearful of making mistakes or causing outages because they want to move fast and take smart risks, but I also don't want a culture where we do substandard work and cause outages that are easily avoidable,” Pham wrote.
> He continued: “Feeling defensive, or feeling like a victim, is NOT the way to get ourselves better.”
I find Pham's response to criticism about the blameless post-mortem to be lacking in self-reflection. The entire point point of a blameless post mortem is to ensure that people don't feel defensive or feel like a victim. To get criticism about failing to follow a blameless post-mortem process and respond by saying: "don't feel like a victim?" tells me that they haven't internalized the criticism in any way.
>The on-call engineer received/acknowledged three alerts about master database being low on disk space, but ignored it. This is not acceptable,” Pham wrote in the email, sent to more than 3,500 employees and obtained by BuzzFeed News. “We are looking to determine whether this is negligence or whether a different on-call engineer could have reasonably missed the alerts amidst a flood of other alerts from the systems at that time.”
I think the timing of the statement was bad - it should have been a result of the post mortem if it was negligence, and not so explicitly stated publicly going in, if only because it can tarnish someone's reputation even if it turns out not true.
There's no question of someone operating in bad faith in the article, though. Pham has a concern about negligence, but that is not bad faith. An engineer can be operating in good faith and be negligent.
I think the emails go beyond bad timing. They clearly indicate ahead of time that the post mortem will be "negligence or a root cause that would have effected a different engineer".
That's not a blameless post-mortem. That's the opposite.
As an employee who has been on-call for production systems for most of his career, this smacks of someone who's never been on-call and is indeed looking for a scapegoat. From the article:
> “We are looking to determine whether this is negligence or whether a different on-call engineer could have reasonably missed the alerts amidst a flood of other alerts from the systems at that time.”
The key phrase is "flood of other alerts" - to me, this sounds as if other shit was hitting the fan and the engineer rolled the dice on letting this sit while fixing something else that was perceived to be more important. Unfortunately, that seems to have been a bad move.
However, it's an eminently understandable move (and indeed, one I've made myself). How much disk is left? What's the rate of change? Can you even know that immediately while still trying to deal with the "flood of other alerts?"
This sounds like there's a lot of things went wrong here, but I doubt the engineer was one of them.
The key phrase is "flood of other alerts"
In 2005 the entire power grid for the clean, famed and punctual Swiss Federal Railways shut down:
http://news.bbc.co.uk/2/hi/europe/4121072.stm
Yep, exactly that. Every train on the entire network shut down due to an overload of the grid. Probably the only disadvantage on being on a 100% electrified network.
It would have been avoidable, the errors where displayed on the relevant consoles, drowned by 100s of other irrelevant errors and warnings and thus not noticed.
Blaming it on the guy who should have made sense of it all and essentially throwing him under the bus in a corporate wide email just about shows what a shitty company Uber really is.
I understand the freakout, i assume it could take down the site. But, like, not responding to a page is like the very last on the list of things to fix. Calling in random engineer is pretty much last ditch, hail mary, the world is on fire, oh fuck we're going to go out of business disaster. At a certain point, you kind of have to assume one possibility is your oncall person is dead. Whatcha gonna do then?
I seriously doubt all of their careful high availability planning failed. I would bet their paging is (hopefully was) just stupid.
I was in the post mortem for this event, the alert for database disk space was piled in to 500 other alerts incorrectly assigned to a rotation notorious for being the most painful in the company. The alerts escalated through to an infrastructure manager and then to an at-the-time director of engineering, none of whom acknowledged the alert themselves.
The alert itself was also of the "disk is 90% full" variety rather than "at rate of growth the disk will be full in an hour and by the way this is the master db". The former wouldn't be a need to panic as you'd have at least a day to resolve it, unless someone is back filling large sets of data.
I've been on many on call rotations for many different projects. Some of them have had such great ideas as "you'll learn a lot from being on call, so we'll add you to the rotation (on your first day)."
Sadly, people don't normally document code and emergency procedures nearly as much as they should. Unlike an airliner, there's no emergency manual sitting around. Even a normal manual would be hard to find, and who knows if it's current.
One place I worked had solved this problem by having an actual operations team, a practice that seems to have faded with time. Our operations team was great. They were there in 3 shifts, 24/7, to cover your ass. They knew more about how the service ran than any dev. There were standards for documenting procedures, and if you got the call (there wasn't an on call rotation, you were only called if everyone was clueless and you wrote it), it's because things were really bad. This was before "devops" was minted the silver bullet.
While a dedicated operations team can save you, and your reputation, most organizations would rather not pay that, and push that cost onto the devs, and make them "feel the pain" of their mistakes.