Hacker News Reader

Comment by t0mas88 | Hacker News Reader

I can't ignore the feeling that Google Cloud is sub par compared to AWS. How did this again cause a multi zone failure. Why haven't they fixed those dependencies the last few times they had a full region failure.

View on HN

zones and regions have different definition in google cloud than AWS. Multiple zones are physically co-located and are not truly availability zones because the physical proximity causes shared fates even when they have independent systems (network, power) that should allow one to fail while another doesn't. Even two datacenters in the same city are prey to the same meteor.

In the same building is quite a bit different than 10km apart, even if a huge meteor would lead to the same conclusion for both scenarios.

Wow, this is a big issue! Is there any way to guarantee AWS-level physical redundancy in GCP without paying the latency/inter-region data transfer (higher than inter-zone outside US: https://cloud.google.com/vpc/network-pricing#egress-within-g...) pricing?

A nice thing about EC2 is that you're getting a pretty dumb, predictable service. There have been multi-zone or global control plane issues but the physical metal has bona fide redundancy between zones/regions.

I don't know what it is like these days but us-east AZs used to be in different datacenters that were on different flood plains and power companies. They were just on a very high capacity (for the day) fiber ring. You still had a couple miles of light-delay latency in between them. A sufficiently big enough meteor, or a powerful enough massive hurricane, could probably take out multiple ones at the same time.

I believe the Google design is one big pool of machines, perhaps spread across a few buildings, but they hope that any failure only affects a few racks.

They will arrange/move workloads such that any one customer will only see an outage in one 'zone'.

Clearly that didn't work here,

That's .... that's not a great strategy. I get that it's cheap, but incidents like this will cost a LOT more with lost business.

> A cluster represents a distinct physical infrastructure that is housed in a data center.

> Google designs zones to minimize the risk of correlated failures caused by physical infrastructure outages

And they have stated that the flood "caused a multi-cluster failure".

> Zones should be considered a single failure domain within a region.

(—GCP's documentation.)

"should be considered" -> we would like you to think

Zones are an artificial construct.

I guess you just discovered the difference between a GCP Zone and an AWS Availability Zone.

By definition, AWS availability zones do not share fault domains, other than a geographic region up to hundreds of miles wide. Even for services used by multiple AZs, such as transit to the Internet and other regions, there are two transit centers operating in separate fault domains.

In contrast, many GCP Zones share the same physical datacenter. What they are actually providing you are simply different racks, rows of racks, or rooms in a single physical facility. Caveat emptor.

Then why do they charge extra for cross-zonal traffic?

Presumably for inter-zonal traffic, it uses up bandwidth on their B4 network, but intra-zonal traffic does not and has basically unlimited bandwidth.

B4: https://storage.googleapis.com/pub-tools-public-publication-...

This is not going through b4 but inter-zonal links are still choke points because they are separate networks

It was not only a region outage, but a global outage!

"GCE Global Control Plane: Experienced a global outage, which has been mitigated. Primary impact was observed from 2023-04-25 23:15:20 PDT to 2023-04-26 03:45:30 PDT and impacted customers utilizing Global DNS (gDNS). A secondary global impact for aggregated list operation failures for customers with resources in europe-west9 has also been mitigated. Please see migration guide for gDNS to Zonal DNS for more information: https://cloud.google.com/compute/docs/internal-dns#migrating... "

Embarrassing.

AWS has had datacenters that were flooded, but they failed over and physically moved racks via trucks. Customers never knew.

This has happened much more than people know.

If it happens without me knowing then they’re doing their job very well.

Well, GCP did fire most of their senior customer techs 18 months ago.

Alphabet is most ruled by arbitrary fiat of empire-builders.

Killing products and footgun changes are standard.

> I can't ignore the feeling that Google Cloud is sub par compared to AWS.

That goes without saying at this point. More importantly, it’s proven worse than Azure.

Depending on the metric. I have not seen any incidents where the isolation between customers was breach. Azure had several. Their compute offerings are better. We could go on.

On the other hand, Azure was(and still is) upfront about not having AZs - now that they have rolled out, hopefully those are not in the same building.

Reply on news.ycombinator.com