Crash-tolerant data storage | Hacker News Reader

Crash-tolerant data storage

news.mit.edu

40 pointsTomte11 comments

I can't remember the last time I lost data to a file system bug. I've certainly lost data to applications running on top of file systems ("Source Safe" -- when they put "Safe" in the product name, you know you have to run away...)

All the stuff I've lost has been due to:

- Hardware failure (vanilla smoke-from-the-disk, or RAID controller failure, or too many RAID volumes dying at once)

- Fat fingers (rm -rf equivalents, or format the wrong drive)

- Data is ancient and in a proprietary format, and the apps that can read it won't run any more (at least, not without a great deal of effort)

- Butterfingers (dropped disk on a concrete floor)

- Media age (drive won't turn / oxide got tired, lost adhesion and rubbed off), or treating media badly (a decade of alternating hot/cold in the garage next to the spiders is not good)

- Vanishing interface standard (no EISA slot, or nowadays, no parallel port), or no drivers for modern OSes

I'm happy there's a provably correct file system out there.

But it ain't gonna save you :-)

> I can't remember the last time I lost data to a file system bug.

You are probably not an early adopter of new filesystem technology. If you were sitting on the edge of technology, you would strongly prefer a provable filesystem.

"The hard drive is ON FIRE!"

https://en.m.wikipedia.org/wiki/Halt_and_Catch_Fire

One fine day I came into work and overnight my Gateway workstation's boot drive had developed 10,000 bad sectors (maybe more, who counts these things?)

Who indeed?

I called Gateway tech support and said, "Hi, overnight my hard drive got over 10,000 bad sectors and I'd like you to replace it, since it's under warranty."

Gateway support: (after going through some really tiresome support nonsense, like power-cycling the machine and verifying a network connection of a computer that can barely boot). "Sorry, we're not going to do that. Your hard drive still has several hundred thousand perfectly good sectors, so it's working fine, no replacement."

Me: (facepalm)

So eventually through some social engineering I got them to replace the drive (a little play acting involving my manager pretending to yell at me, which got instant sympathy from the Gateway support person. This is a useful tool). Really I just went over to Fry's and bought a new hard drive. But it was the principle of the thing.

And I resolvde that the next time I had to call in a hard drive failure, I would say:

"Hello, Gateway? My hard drive is ON FIRE and my computer won't boot any more. What do you want me to do?"

I remember the one I got years ago https://www.youtube.com/watch?v=-e9OcWIH6N8

I've seen data loss due to bugs:

  * ext3/ext4
  * reiserfs (reiserfsck used to eat data *all the time*)
  * xfs
  * jfs (can't remember kernel version, but friend lost tons of data)
  * ntfs

Filesystems I haven't lost data on:

  * ufs, ufs2, ffs -- the BSD ufs variants
  * zfs

Filesystems I haven't done much with:

  * hammer (would be amazing for maildir storage)
  * btrfs (avoiding intentionally)
  * reiser4 (barely used it and then the whole murdering happened)
  * tux (always sounded nice)

There are lots of other filesystems with special use cases, but I'm ignoring those. These are just more relevant general purpose filesystems.

And note that filesystems without journals, logs, or being CoW (ext2, [ex]FAT, etc) I'm ignoring because they aren't designed to handle abuse, power loss, etc.

This paper about linux filesystems is worrying: https://www.usenix.org/conference/osdi14/technical-sessions/...

Yeah, avoiding file systems that don't have any transaction nature is a sound principle.

I was dismayed that NTFS "was transactional" but that those transactions didn't mean that it saved user data, it just made sure that filenames and directory structure was preserved. That's entirely missing the point, guys. (Later, I got a peek at the user-data-level transactional extension to NTFS, and had a good case of the horrors. Run away; MSFT is deprecating it, last I heard, and good thing, too).

Most filesystems default to saving only metadata in the journal. Writing all file data through the journal is much more expensive (unless you are log-structured).

Most of these problems can be averted by a "port early, port often" approach to data storage. Everything should be copied to a new piece of media every five years at most, in whatever the newest stable media format is these days. If that means the old hardware can't read the new media, then it's time to write an emulator for the old hardware that runs on new hardware. That also saves you from the "proprietary format" problem later on—if you have that emulator, you can still start up the app 30 years later in QEMU and get everything exported over the virtual serial bus.

As for fat fingers—that's what creating a filesystem abstraction on top of versioned object storage is for!

That is interesting -- using Coq to prove crash safety and consistency. It is a bit in the spirit of how Kyle (Aphyr)'s Knossos tool checks databases in his "Call Me Maybe" series.

There should definitely be an equivalent thing for data corruption. There sure have databases and systems that made lofty marketing claims only to then silently corrupt user data (really, one of the most terrible things that could happen -- as it will corrupt your backups and might not be noticed for a while).

Also due to proliferation of VMs/containers. The model checker would have to include abnormal behaviors in both the host and the VM instance. Say pull the power on the host or VM.

Looking at products that store data I have both designed my own, so I know exactly how and when data checkpoints are happening or evaluated a few DBs. Or I like CouchDB for an off-the-shelf example. It has an append-only and crash-only design. Stopping can be done by kill -9 the process or backing can happen anytime, take EBS snapshots anytime. You'll pay for it perhaps in raw write performance but it has worked great for me.

Reply on news.ycombinator.com