Hacker News Reader

Comment by Johnny555 | Hacker News Reader

That could be a power surge taking out a whole rack

This failure mode, at least, is already accounted for by sharding data across cabinets:

Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet.

https://www.backblaze.com/blog/vault-cloud-storage-architect...

However, they don't seem to offer multi-datacenter (or multi-region) redundancy so are still susceptible to a datacenter fire/failure.

In comparison, AWS S3 distributes data across 3 AZ's (datacenters), and you can further replicate across regions if you choose. Though you pay for that added redundancy in 3 - 4X higher cost.

View on HN

A better example would be the ceramic bearing fiasco that NetApp experienced with Seagate. Seagate had switched to a floating ceramic bearing on one family of their fiber channel drives. In those drives one or more of the bearings would shatter and start spreading ceramic dust across the disk surface. This happened between 3 and 4 years of run time and the disk would rapidly fail after that happened. People that bought a filer with several hundred drives started seeing large numbers fail suddenly. Up to the point they failed, not any indication. But it was run time hours related and thus correlated to all drives that started running at the same time. If you had put those drives in different data centers on different racks on different servers it wouldn't have mattered if you started 20 at the same time to be your 'tome', when they started failing it was possible to lose them all in a week.

There was also a failure mode in Seagate drives where the bearing increased in stiction. As long as it was spinning, there was no problem. But if you spun it down, it might not spin up again. If you had a group of disks powered up for a long time, many could fail together at the next power cycle.

A chaos monkey that randomly powers down disks one at a time can prevent this.

Though that sounds like a fixable failure mode, so it's more annoyance than tragedy.

You can tell from the reports they put out that Backblaze doesn’t put all their eggs in one basket, they use different brands and drive types.

I thought AZs were in the same physical location, just separate networks, no?

No, they're different physical locations.

From https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Conce...:

> Each AWS Region has multiple, isolated locations known as Availability Zones.

And from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-re...:

> Each region is completely independent. Each Availability Zone is isolated, but the Availability Zones in a region are connected through low-latency links.

Nope. They aren’t even in the same city. For example, each of the AZs in us-west-2 are separated by about 50-60 miles.

This doesn’t list all the locations, but is a good map to get an idea:

https://www.google.com/maps/d/u/0/viewer?ll=50.9584270000000...

That map is incorrect: us-west-2a, 2b and 2c are not static names for the AZs. Every user gets their own mapping of which physical location is a, which one is b and which one is c. My us-west-2a may be your us-west-2c. They are not the same.

Extremely interesting, thanks for this clarification. For those curious, it's documented here[0] -- search for "To ensure that resources are distributed across the Availability Zones for a region"...

[0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-re...

Yes, I know they pseudo randomize the allocations. That is irrelevant to my core point, that each AZ is not just separate networks in the same building or even adjacent buildings, but rather they are truly isolated by a non-insignificant distance of somewhere around 50mi on average.

It's a cool map, but I can't find any reference for the source of the data center locations.

AWS purposely doesn't publish that information, and while I can believe it's possible to crowdsource the data by doing a little sleuthing (or working for certain vendors), it's hard to trust the map without knowing the sources.

There’s a The Atlantic article[0] that explains how to source the data pretty easily if you are willing to put in the effort.

https://www.theatlantic.com/technology/archive/2016/01/amazo...

They are separate datacenters, I don't think they make any promises about how far they are apart from each other, but at least in some regions they are 10+ miles apart.

The AWS Cloud infrastructure is built around AWS Regions and Availability Zones. An AWS Region is a physical location in the world where we have multiple Availability Zones. Availability Zones consist of one or more discrete data centers, each with redundant power, networking, and connectivity, housed in separate facilities

https://docs.aws.amazon.com/aws-technical-content/latest/aws...

So, a single AZ could span across multiple physical locations, as well?

Am I reading that correctly?

Others have answered, but I think the general principle for a Region is that between the AZs you have less than 1ms latency, physical separation between the data centers, but they may still be in the same floodplain, be able to be hit by the same hurricane, etc.

More info here if you want to see how regions are structured at a high level of detail: https://youtu.be/AyOAjFNPAbA?t=15m21s

Multi-region redundancy was on the roadmap for their B2 offering 10 months ago, not sure if it has shipped yet.

https://news.ycombinator.com/item?id=15125643

I poked around their pricing page a bit before I posted to see if I could find multi-site redundancy options, but couldn't find anything.

Yev from Backblaze here -> Not out yet, but we're working towards it!

Reply on news.ycombinator.com