Hacker News Reader

Comment by dekhn | Hacker News Reader

zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.

View on HN

> A cluster represents a distinct physical infrastructure that is housed in a data center.

> Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages

And they have stated that the flood "caused a multi-cluster failure".

> Zones should be considered a single failure domain within a region.

(—GCP's documentation.)

"should be considered" -> we would like you to think

Zones are an artificial construct.

I guess you just discovered the difference between a GCP Zone and an AWS Availability Zone.

By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains.

In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.

Wow, this is a big issue! Is there any way to guarantee AWS-level physical redundancy in GCP without paying the latency/inter-region data transfer (higher than inter-zone outside US: https://cloud.google.com/vpc/network-pricing#egress-within-g...) pricing?

A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.

I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.

I believe the Google design is one big pool of machines, perhaps spread across a few buildings, but they hope that any failure only affects a few racks.

They will arrange/move workloads such that any one customer will only see an outage in one 'zone'.

Clearly that didn't work here,

That's .... that's not a great strategy. I get that it's cheap, but incidents like this will cost a LOT more with lost business.

In the same building is quite a bit different than 10km apart, even if a huge meteor would lead to the same conclusion for both scenarios.

Then why do they charge extra for cross-zonal traffic?

Presumably for inter-zonal traffic, it uses up bandwidth on their B4 network, but intra-zonal traffic does not and has basically unlimited bandwidth.

B4: https://storage.googleapis.com/pub-tools-public-publication-...

This is not going through b4 but inter-zonal links are still choke points because they are separate networks

Reply on news.ycombinator.com