Hacker News Reader

Comment by m0zg | Hacker News Reader

99 cases out of a 100 it's a "configuration change" in a very large system that's so large and complex nobody can comprehend it anymore. Decentralize, folks.

View on HN

Not sure why this gets downvoted. This is an entirely valid comments.

It's getting increasingly obvious that despite being highly distributed, cloud systems run by Google, Amazon and other big cloud companies are inherently centralized. If nothing else, they have centralized controls.

A decentralized system can, of course, fail, but it would never have "configuration change" as the root cause.

S3 is (for want of a better term) a legacy system in AWS, having been around since pretty much the beginning. A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global).

Beyond those bucket mutations, everything else is purely regional scoped.

I'm _really_ surprised to see that GCP has made the same mistakes though and had things tied together. I'm not convinced there's a good excuse. AWS/S3 is understandable. No one knew what things would look like. Not one engineer would have made the same decisions then if they'd known what the future was for the platform. Google came to the cloud market much later. Their engineers should have seen the issues with AWS and the problems that interdependence cause, and specifically architected isolated infrastructure (which from this post-mortem it seems like they didn't). That's both remarkable and disappointing to see.

GCP may have come later than AWS but googles internal blob storage could of predated AWS for all we know.

>"A series of choices made when the idea that there would ever be more than two regions, leave them unfortunately stuck having to tie regions together to a limited degree (bucket names are global)."

Could you elaborate on those choices? Also what do you mean by they're "stuck having to tie regions together"?

The main thing is that bucket namespace is global. There is only one bucket "foo" worldwide. That means that every bucket mutation event (e.g. bucket deletion or creation) requires co-ordination between regions. When that big us-east-1 region outage occurred a year or so back every other region worked fine except for bucket mutations.

The worlds infrastructure is inherently fragile. It should be difficult to take down these large distributed systems, but they are over-coupled.

It's very easy, apparently, to take them down by a configuration change. Happens several times a year and takes hours (and untold millions of dollars in productivity loss) to recover from.

Sure it can. Let's back up a bit to something physical as an example. If every car on the road were self driving, that would form a decentralized and distributed system. Each car independently monitors and reacts to its surroundings. That doesn't prevent accidents between two cars, nor does it prevent huge cascading accidents that can cause highway closures.

Decentralization often makes these scenarios harder to avoid, not easier. There are simply more ways that things can interact (and therefore break).

To avoid these failures in any setting, you need to have provable containment of failures while assuming part of your network is malicious. That's asking for a lot more than decentralization.

Like I said, a decentralized system can fail, but the root cause of such a failure cannot be "configuration change". If a configuration change is sufficient to break your system then it's not truly decentralized.

Let's take your car example.

If you update your car's firmware and get in an accident, it's not a system-level failure.

If manufacturer sends a wireless update and all their autonomous cars start to bump into one another, it's not really a decentralized system anymore - updates are centralized.

If you update your car, it crashes, sends a wireless signal that affects another car, and it crashes and so on... This is an example of a cascading failure in a fully decentralized system, but the real root cause would be cars interfering with one another, not your firmware update.

If a manufacturer with 10 percent market share sends out an update that causes their vehicles to suddenly turn right, that would ruin conditions for everyone.

If that's the case, then the conclusion based on the upthread reasoning is that this system isn't decentralized enough when any single manufacturer has a 10 percent market share.

It prevents all cars failing at the same time worldwide.

Unless you push out a bug that is triggered by some combination of events so distant from the release that you could never have caught it in testing. Perhaps a bizarre interaction between multiple unrelated systems from different vendors that all work tested in isolation and only fall over when a series of 1 bits in a certain pattern in the real time clock chip creates a blip on a bus that gets corrected in a way that was interpreted differently by two different teams and the fallback system tries to take over but the primary system comes back online at just the wrong time and trips a bus contention bug that was never seen because testing never hit this perfect storm of race conditions and, well, the front falls off.

The point of me making up a stupidly improbable story is that stupidly improbable events keep happening despite all the clever people in the world. And of course the people on hacker news who know even better.

Decentralized portions of an architecture must have pathways between them. These pathways then contribute their own points of failure, both in isolation and in combination.

A lot of these kinds of systems are decentralized. It's not a magic bullet.

It's not that simple. I'm not an expert at gurangutan scaling and I'd bet you're not either.

I am. I used to work at Google and on Cloud in fact. Fact of the matter is, 99.99% of Cloud users don't need "gargantuan" (I'm assuming that's what you meant) scaling. Most customers don't need terabyte-scale anything, let alone petabyte-scale anything. Their stuff could run just as well (and more reliably) on an isolated instance that Google SREs can't mess up. And don't get me wrong, this is not a dig at Google SREs -- they're far and away the best in the industry, but configuration of large systems is a de-facto SPOF.

Reply on news.ycombinator.com