I imagine a room full of Amazon engineers all standing around freaking out and possibly yelling at interns.
I imagine a room full of Amazon engineers all standing around freaking out and possibly yelling at interns.
The first and most important principle that every call leader promotes is "stay calm". It would be natural for folks to freak out and get stressed when the clock is ticking on an outage. There's an innate human sense that our emotional state has to match the urgency of a problem. But this would only lead to a kind frenetic energy that isn't helpful.
Instead, maintaining a collected composure quickly cools everything down so that actions can be thought through and communicated more clearly. Many people are surprised that when they join a call, especially with an experienced team, that it is so calm. If you listen to the audio of Michael Schumacher as an F1 driver, or to fighter pilots performing complex maneuvers, it can also be striking just how calm they are. It's very effective and a great lesson for teams under pressure.
In that situation, and especially at scale, it's very useful to keep everyone on the same page about what actions are happening when, and to help folks out by showing them how to find things which may have changed (keep in mind the change may have been in a totally different system than the one that ultimately breaks). For example: it would slow things down if someone tried to flip data centers at the same time that someone else was rolling back software.
If on a call it becomes clear that a team needs to make some kind of non-rollback change, like a code change, or push a new config, and thankfully that's very rare, we'll generally recommend they break off and work on it in isolation, while nominating a point person to liaise with everyone else. We do use chat too and that can be more effective as more people join.
For postmortems we try hard to do in writing first, and then an in person review. It's very helpful to be able to re-assure and guide each other. I can back up the "Blame the system, not the engineer" guidance others have written about here too.
In a microservices or SOA architecture, services can layer on top of other services, that call more services, that put messages in a queue for handling by yet another service, etc., belonging to different teams. Unwinding this to identify the problem is something that's much faster with a phone call. "Connecting the dots", especially across teams. A technical discussion within a team is likely to take place by chat.
I've participated in a number of tech conference calls over the years, and I've always been impressed with the professionalism: the discussion is calm and focused on identifying the problem and recovering. I think folks strive for the "cool radio voice", the Houston Center Voice like was described in the SR-71 Ultimate Ground Speed Check story: http://oppositelock.kinja.com/favorite-sr-71-story-107912704...
> Now the thing to understand about Center controllers, was that whether they were talking to a rookie pilot in a Cessna, or to Air Force One, they always spoke in the exact same, calm, deep, professional, tone that made one feel important. I referred to it as the " Houston Center voice." I have always felt that after years of seeing documentaries on this country's space program and listening to the calm and distinct voice of the Houston controllers, that all other controllers since then wanted to sound like that, and that they basically did. And it didn't matter what sector of the country we would be flying in, it always seemed like the same guy was talking. Over the years that tone of voice had become somewhat of a comforting sound to pilots everywhere. Conversely, over the years, pilots always wanted to ensure that, when transmitting, they sounded like Chuck Yeager, or at least like John Wayne. Better to die than sound bad on the radios.
There are still areas of Canada (Southhampton Island) and Central America (Panama IIRC) that are on EST year round.