Pocket Watch: Verifying Exabytes of Data

Yeah to be clear my point from the article is that even though you can model an incredibly high durability number in theory, there's a lot more work required to see that in practice. I'm sure the S3 team also have extremely high internal durability numbers.

A lot of other factors go into an end-to-end durability numbers, including the possibility of software bugs, correlated failure, etc. Magic Pocket has an external durability claim of 12 9s which gives us a lot of headroom between the external figure and the internal model.

Fortunately both 11 or 12 nines is close enough to infinite as far as clients are concerned.

27 9s is literally higher than my confidence that human civilization will be here in the next second. 10^27 seconds is about 32 quintillion years. Extinction events occur with a much higher frequency than that.

No criticism of Dropbox here; they know that number is just math games, too. I'm just putting some numbers on how true that is. Because I find tossing about big numbers like this as if they mean something a bizarre, nerdy sort of fun.

S3's durability numbers are for a single region (since they were quoting that number before they supported cross-region replication) -- so you'd get much better durability if you mirror your objects across regions. Which, apparently, is what Dropbox does.

Though if you require such high durability, you're better off with multiple providers in multiple countries, since at 11 9's, it probably becomes more likely that a provider will go out of business or a civil disturbance will make your data become unavailable than the chance of them losing it.

From what I hear from other major storage providers Magic Pocket is definitely very high up on the paranoia/verification scale. There is of course a cost to all this verification traffic. Any sensible storage provider will be doing stuff like disk scrubbing and verification scans on their storage index however.

We haven't shared drive reliability stats before but I don't think we have any objection to doing so. Don't expect these in the next month or two but we'll likely get to this in the not-too-distant future.

>> Was shocked to see that your durability (27 9s) was so much higher than what S3 claims (11 9s)

> 27 9s is literally higher than my confidence that human civilization will be here in the next second. 10^27 seconds is about 32 quintillion years. Extinction events occur with a much higher frequency than that.

Without reading the article, I'm assuming this is the uptime probability they promise - so ( 1.0/10^9 ) * 365.25 * 24 * 3600 * 300 ~ 1 second of unavailability every 30 years (AFAIK Amazon is pretty far in the "red" on this one - they've had a few outages?) [ed: initially was off by a factor of 100 (and 10 in error...), the two 9s before the comma - the "compliment" (1 - p) of 11 9s is 1/10^9, not 1/10^11)].

11 9s is already effectively the same as will "never go down in a way customers will notice" (actually a second every 30 years per region isn't entirely insignificant, only almost insignificant). A higher guarantee does indeed seem silly. It's probably much more likely that we'll see annihilation by global thermonuclear war (for example). In which case I'm guessing the data would go off line for a while -- so such a number is meaningless.

Yup we do this, and yes, the verifier has been broken before (during early development, not at production scale).

We do a number of things here, like taking down a sufficient number of storage nodes in a single region to make blocks appear "missing" in the region and force an automatic failover to the other region (this is transparent to users, apart from slightly more latency), or more direct/risky checks in our Staging cluster (we don't ever mess with data in our main production cluster).

In reality a large system like this regularly encounters timeouts or failures of sub-components which are masked by our multi-region redundancy but show up as spikes in the verifiers. These remind us that everything is working, in between more explicit DRT (Disaster Recovery Training) tests.

Thanks for the feedback! Glad folks are reading this stuff :)

> Would you be able to give more details as to how you use disks as raw block devices, bypassing the filesystem? For example, the kind of abstraction layers you use, their interfaces and methods, disk layout of extents, metadata layout, consistency issues, fsync issues, and the various performance advantages from bypassing the filesystem?

We can definitely go into details on the raw block device. Will probably leave that to @jamwt to write a blog post on since there's quite a lot of content there. Most of the motivation for doing so was actually so we could exploit the write-zone layout of SMR disk drives, although we get some other minor benefits like a bit more "formatted" disk space and no filesystem bugs.

> I was also wondering, I know you have two functional zones, but do you also keep an isolated timelapsed append-only backup of all data, in case of ultimate software error (something bad gets past the verifiers, or a delayed bug kicks in)?

Quick summary is that we write typically only data into at least two zones (regions), obviously replicated within these zones, and then we keep "trash" around in case there's a software bug that causes deletion. We also have a delete grace period where we keep data within the storage system for longer in case an application issued an erroneous delete.

The Dropbox filesystem layer actually checks that a file was added to MP before committing it to the metadata layer of the filesystem, so there's an extra check there that the write didn't just disappear.

We also log metadata about each put and every internal data transformation to a separate system so we can retrace our steps if there was ever an issue. This logging system is actually running on HDFS, so MP isn't the only storage system running at Dropbox, but it's by far the biggest one.

> Would you be able to give more details as to how you use disks as raw block devices, bypassing the filesystem? For example, the kind of abstraction layers you use, their interfaces and methods, disk layout of extents, metadata layout, consistency issues, fsync issues, and the various performance advantages from bypassing the filesystem?

Yeah, we're going to write a whole blog post on this sort of stuff sometime soon. Needless to say, it's a big topic!