Hacker News Reader

Comment by teraflop | Hacker News Reader

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ...

This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007:

> In closing we point out a challenge that we faced in testing our system for which we have no systematic solution. By their very nature, fault-tolerant systems try to mask problems. Thus they can mask bugs or configuration problems while insidiously lowering their own fault-tolerance.

As developers we can try to bear this principle in mind, but as Monday's incident demonstrated, mistakes can still happen. So, has anyone managed to make progress toward a "systematic solution" in the last 9 years?

View on HN

Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole.

The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, every year Google does DiRT [1] exercises which test system responses to a large calamity, eg. a California earthquake that kills everyone in Mountain View and SF including the senior leadership, and also knocks out all west coast datacenters. The half-life of code at Google (in my observation) is roughly 1 year, which means that half of all code has never gone through a DiRT exercise. The same applies to other, less serious fault injection mechanisms: they may get executed once every year or two, and serious bugs can crop up in the meantime. Automated testing of fault injection isn't really feasible, because the number of potential faults grows combinatorially with the number of independent RPCs in the system.

I'd be willing to bet that the two bugs that caused this outage were less than 6 months old. In my tenure at Google, the vast majority of bugs that showed up in postmortems were introduced < 3 months before the outage.

[1] http://everythingsysadmin.com/2012/09/devops-google-reveals-...

Direct link to DiRT => http://queue.acm.org/detail.cfm?id=2371516

I remember my founder (ex Googler) telling us about fault injection at Google. We were pretty amazed by the idea. Thanks for the link @nostrademons.

Testing doesn't detect failure, it only detects the failure of a test. Real failures happen more often than test failures, for the same test on the same code with the same input and output. The best systematic solution would detect real failures, not see what happens when you fail a test.

That's monitoring, then. As Steve Yegge's Platforms Rant [1] mentioned, testing and monitoring are two sides of the same coin. Google does both, but the original thread-starter here was asking about how to detect failures when the system itself is designed to mask & recover from failures. (FWIW, most such systems do log when they've encountered a failure condition and recovered from it, and this stat is available to the monitoring system.)

[1] https://plus.google.com/+RipRowan/posts/eVeouesvaVX

Basically, yes. But we don't have to make a traditional monitor, or have it be an extra component. Monitoring all the facets of, say, a code deployment, or a software build, or performance testing, is a dynamic thing. It may fail, or it may succeed, or it might be suspicious.

Normally we design systems for humans to determine that 3rd part; in this case, there should have been a system where humans could see the one or two pieces of unusual activity and investigated. But there wasn't, or it didn't work right. So a "fix" would be to develop software that adapts to nondeterministic behavior the way a human does. I wouldn't exactly call that monitoring, though.

There was an relevant section in the Google SRE book notes posted here the other day about injecting faults into Chubby, their distributed lock service, which was too reliable.

Ex: Chubby planned outages Google found that Chubby was consistently over its SLO, and that global Chubby outages would cause unusually bad outages at Google Chubby was so reliable that teams were incorrectly assuming that it would never be down and failing to design systems that account for failures in Chubby Solution: take Chubby down globally when it’s too far above its SLO for a quarter to “show” teams that Chubby can go down

This is an interesting question, and seems to get to the core of Nassim Taleb's ideas [1] about fragility and the limits of what we can understand, and how many of our attempts to create artificial stability ultimately bring about the opposite.

That said, based on this post-mortem, I think Google, and our industry as a whole, is doing a pretty good job. Periodic failures like this are inevitable, and if they serve to make it less likely that a similar failure occurs in the future, then that is a system as a whole that could be described as "anti-fragile".

[1] At least my interpretation of them

> So, has anyone managed to make progress toward a "systematic solution" in the last 9 years?

That depends on how you define "solution". If development time isn't a concern, then formal verification is a pretty solid solution. AWS has used TLA+ on a subset of its systems. [0]

[0] https://en.wikipedia.org/wiki/TLA%2B

The standard solution in realtime safety-critical systems is to perform health monitoring in addition to robust fallbacks, such that when the system is falling back, it is reported as unhealthy.

For example, the CAN bus normally has an automatic retry feature on a variety of errors. A properly functioning CAN bus should have a bit error rate that is nearly zero. Lightly loaded, it can tolerate a very high error rate (say, due to noise, poor termination, etc). In that situation, the product would report a specific warning message to higher-level SCADA systems, such that it gets bubbled up all the way to the operators.

The approach at Google is to report the actual error rate up to the monitoring system, and then let the monitoring system decide at what threshold to alert with a warning message. This lets you catch a wide variety of errors, eg. if a single replica has a high error rate, that's probably a wildly different problem from if a whole rack of machines has a high error rate, which is different from every machine in the service having a high error rate, which is different from only the set of machines that were fed a specific query having a high error rate.

One of the bugs in this postmortem was that the process in question didn't do this, instead masking the error. Somewhat understandable, as I found the whole "execute a fallback, report the failure, and let the monitoring rules deal with it" philosophy one of the most confusing parts of being a Noogler. If you've never worked on distributed systems before, the idea that there is a monitoring system is a strange concept.

CAN and it's automatic retransmit is actually pretty good example of how simple transient problems can quickly overgrow into global system failures. On typical CAN the bandwidth headroom is small enough that when all colliding/failed telegrams would be blindly retransmitted the collision rate would skyrocket and thus only high-priority traffic will make any progress, and as on CAN priority and purpose is intrinsically linked together, from the global point of view nothing will make progress. That's why most CAN controllers have configurable retransmit behavior per packet (drop/retry/raise error and application deal with that) and partially why today's cars have multiple CAN buses.

The best example so far regarding fault tolerance. The Netflix Simian Army. Introduce failures constantly!

http://techblog.netflix.com/2011/07/netflix-simian-army.html

> So, has anyone managed to make progress toward a "systematic solution" in the last 9 years?

Allow me to introduce you to the fantastic and battle-tested http://learnyousomeerlang.com/what-is-otp , preferably utilized (IMHO) via http://elixir-lang.org/

Degraded modes of operation is one example of how to visualize masked errors. Another is to trigger an alarm on fallbacks.

As a general reflection, many distributed system leave out the cause of their changes and only log actions. Instead of logging "new membership, new members are b,c,d" you are better of logging "node a has not responded to heartbeat in the last 30 seconds, considering it faulty". Following such a principle makes it much easier to spot masked bugs, since you can reason about the behaviour much better.

Aggregating logs to a central location and being able to analyze global behaviour in retrospect is also a great feature.

Great point! More precisely, each state transition in a system should report the old state, the new state, and the triggering event (cause) to a monitoring system (possibly just a log).

Reply on news.ycombinator.com