Hacker News Reader

> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes.

In another post mortem by Google I read that Google engineers are trained to roll back recent configuration changes when an outage occurs. Why wasn't this done this time?

View on HN

The literal next paragraph:

> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage

Maybe they did rollback and that took longer than the target time (for whatever reason). But it's hard to know since this post mortem is fairly vague.

Not just maybe. It says quite clearly that the network was overloaded and, as a result, their configuration changes took too long to arrive at the affected components.

Reply on news.ycombinator.com