Hacker News Reader

Comment by cwillu | Hacker News Reader

“Oscar announces, “I’m seeing some log entries from the web server that look a little weird. I’m gonna look at those.” This is the beginning of a 5-minute silence. […] So it’s incumbent on you to interrupt this silence.”

This is “we need to do something, this is something, we need to do it” thinking. The role of the commander imo is to insulate the investigators from exactly this sort of meaningless interruption.

““I need 5 minutes” [...] There is, however, a healthy kind of dead air.”

If you need to be told this, you are being managed by your staff, not managing them.

View on HN

The problem is that, as incident commander, I don’t know the difference between “I’m not saying anything because I’m stuck” and “I’m not saying anything because it’s going to take me five minutes.”

The correct rule of thumb is to always over communicate, regardless of your role. If you’re troubleshooting, tell the incident commander that you’re doing X and it’ll take around 10 minutes before you see results. Then you’ve set expectations.

It would be sort of unprofessional I guess, but it seems like it would actually help if people could sing mindless tunes as they work. If somebody is going “do-do-do, doot dadodo” as they work, you know they are in progress. A sudden “hmm” or silence indicates trouble.

But that's not over communication, that's just pollution of radio space.

For example, in a recent incident I ended up kinda in charge / responsible for the database as the application encountered some weird lifelock with row and table locks. In such a situation, if no one needed the radio space, I ended up announcing the status of the database every few minutes, even if nothing really changed.

Or, if someone does something, give feedback if this has a noticeable effect on your system or not. Or quickly ask if you want to do something that could affect them, no matter how little that is. "App, I'll modify setting xyz, tell me if that has an effect".

>I ended up announcing the status of the database every few minutes, even if nothing really changed.

Yes exactly, in a incident if there is some long task that is running that will clear the incident, this is important.

It's essentially a progress bar.

aka The Everything's Okay Alarm

That's the sort of anxious micromanagement a good manager is shielding their team from in a situation like this. You need to trust your guys. They're the experts.

This is all in the context of a incident and the role of a IC.

The author nails all the issues here. Dead silence with no shared screen is not good for a incident (without any other context).

You either communicate constantly on what is happening or say "I need 10 minutes to debug" In which case 10 minutes is given but they must come back with an update at that time.

an IC absolutely cannot just let someone go off and trust they are debugging the issue.

Why? To any of this?

If you aren't fixing/investigating the problem, why get in the way of the people who are on the basis of trust issues, or some ICs idea of what's "good for a incident"?

How do you know you are getting in the way, how do you know the guy just didn't go out to lunch? How long do you wait to hear back, do you just wait forever?

The scenario I have in my head is a conference, the person in the example has no video, no shared screen and no response.

The 5 minutes is just an example, maybe a bad one but the idea still holds.

> how do you know the guy just didn't go out to lunch?

You don't, you hire people you can trust, and with whom what is/isn't appropriate during an incident is clear.

> How long do you wait to hear back, do you just wait forever?

A short message "How's that going?" if it seems to be taking a long time and there is no communication.

What you are describing is micromanaging the actions of the people who are meant to be experts in that domain. Why is this needed? What issues have arisen before, and do these issues arise during normal development.

> person in the example

It's not clear to me the role the people in the example have.

What does being "primary investigator" mean? How does that relate to Ops - why isn't support, or the devs investigating the logs. Who in the example would be knowledgeable about this area? Is the Ops person qualified to know if a web log is strange? Why would they be pushing fixes to a web server?

To me, if people in the know are competent enough to investigate an issue, they should be competent enough to communicate about it.

The incident commander isn't necessarily a manager. Indeed, where I work it would be uncommon for a manager to take incident command and unlikely that anyone above the first level of management would join the incident call.

Not at all. The way you gain trust is by making clear commitments and either meeting them or updating them as necessary.

I'm not saying that the engineering troubleshooting a problem should be narrating moment by moment. I'm saying they should say, at the outset, something like:

"I'm going to go try flushing the cache and restarting the app servers. I expect this to take about fifteen minutes, and if it works I would expect to see the database load normalize."

That's all I need as an incident manager. Otherwise I'm sitting there not knowing what to say in my status updates, my stakeholders are asking questions about estimated duration that I can't answer, and so on.

And as an example, since we've been dealing with a huge amount of incidents over the last few weeks: To me, it's completely normal if BryantD is like "Yo tetha, how are the caches and restarts coming?" and it doesn't take much for me to mumble something like "16 out of 28 done. database, you see anything changing?" No need to be vastly eloquent or anything.

Or I might throw something like that in on my own if things are dead silent right now and no one else needs the radio. And I think that's a good habit other people on the team are picking up. If someone needs the air for something important, they can talk, but otherwise you have various status updates floating around. And more often than not, those result in "Wait, did you just say xyz changed? Did I just do something useful?"

I see we'd enjoy working together. ;)

Out of curiosity, how do you parallelize incident response? I am fairly picky about only changing one thing at once but I'm not sure I'm always right about that.

To me, this depends on the different failure domains and how they interact with each other and based off of that, it can be decided if the teams should coordinate changes, communicate changes or just run.

For example we've had situations in which... creative user and amazing code managed to fry both the dedicated elasticsearch cluster for the application as well as the dedicated database, and the application servers were also weird. Here, we'd split into three small teams each responsible for each of these pieces.

And looking at these failure domains - ES and postgres don't interact with each other, so both of these teams should just run independently as fast as they can to get their components working again and inform the app team when they are back up. However, the application team should closely coordinate their actions with both of these teams - I've had enough situations in which someone pressed the "hilarious load on ES" button while people were still getting ES up to capacity... and down we go again.

Or in a similar way, we had a central database outage and a couple dozen apps got taken out. Database in the lead, sure, but once that's going again, the different application teams can run free and make changes with loose coordination with the database team.

However, within the same failure domain, I really don't like to make too many changes in parallel. Getting back up ASAP is a priority, for sure, but what about the outage in 2 hours, or tomorrow because we just hit the system with a couple of wrenches and have no idea why it's back up? Here I strongly prefer deliberate, individual changes so we can get an idea why the system failed. 10 more minutes of downtime / degraded service now can safe us many nerves and downtime hours over the next few days.

That makes sense. I am slightly envious of the idea of having clearly discernible failure domains but that’s a me problem.

The coordination can get hairy at a huge company but that’s why (for example) Amazon has a really challenging program for becoming an incident manager, which creates a pool of people who can understand failure domains quickly for incidents which span a lot of the stack/company.

This is a constant architectural struggle, I tell you.

Like, inside the infrastructure, I'm constantly updating, poking teams and such to make sure we have clean disaster recovery layers and we have clean documentation about the dependencies between our infrastructural services. And to make sure we don't have cycles - or at least we should have good documentation about these cycles and how to handle them. And to make sure your services don't intermingle too crazily. We should have our postgres bubble, and this should depend on the consul bubble through a clean interface.

And similarly, I'm constantly telling our dev-teams that even though we have micro services and everyone is entirely free to do whatever they want within their team (within reason), we should have clean boundaries between these services. Clear interfaces at an HTTP, AMQP and gRPC layer - and dependencies between services maintained by different groups of developers should go through these dependencies.

If you want to share something that's currently an internal service of some application or system, don't just depend on it because you can - that will get annoying for /both/ teams. Rather separate it out cleanly into a new service with a defined interface, and share that. Our infrastructure should be a DAG of small-ish, independently fixable, deployable and functional bubbles, not a huge ball of mud with everything going everywhere.

It's a constant struggle, but I think it improves both our architecture at a development level, overall dev experience because of better separations of responsiblities and our stability because we can structure incident responses better. Thank you for reading my ted-talk.

Yeah this article didn't go where I expected it to go. Silence is nearly always an indication that someone is doing something or thinking about something, and the advice here is to interrupt them?

But how do you know that without any other information?

In a high stakes incident you cannot let radio silence go off without clear communication on status updates... this was called out in the bottom.

You train people, delegate to them, and then trust them to do the right thing.

This is discussed in the article as well, but I think expectations are much better than interruptions. And even then, it is better to set those expectations during training rather than during the response. People should be trained to prioritize communication during a response, but not above their work doing the response. And the people doing the work are individually best suited to make the call of whether what they're doing is more or less important than communicating.

In my view, it's really hard to overstate how hard it is to be reading big volumes of logs, reconstructing the runtime state of some big complex distributed system from the breadcrumbs available, and thinking about what to do next to most quickly mitigate or get more information on the issue, while being bombarded with messages.

I read that part as meaning this probably doesn't need to be a call at all at this point in the investigating process.

Reply on news.ycombinator.com