Hacker News Reader

Comment by UltraViolence | Hacker News Reader

I'm surprised that no file system has made it into mainstream which has error-correction capabilities (without RAID) such as Reed-Solomon or Fountain Codes.

Also, I'd like to see file systems with cryptographic hashes for protection instead of checksums.

Surely these would be extremely useful for long-tem storage formats?

View on HN

My impression (not based on anything firm) is that basic reasonable-length CRCs are extremely unlikely to have collisions, unless someone is deliberately trying to create collisions. If I were looking for insurance I'd turn to one of the fast 128-bit non-cryptographic hashes.

I used to download movies and stuff from Usenet and I noticed that there were a lot of errors in the files even though TCP/IP uses checksums. I assumed this came from people downloading and uploading the files repeatedly and errors creeping in.

That's why these days I always use BitTorrent, because it uses cryptographic hashes (albeit the outdated SHA-1) and I'm therefore guaranteed that every bit is identical to the source.

The problem isn't the checksum algorithm, it's that TCP/IP only protects packets in-flight. The checksums aren't kept on-disk, so errors can creep in after they've made it across the network.

Well, and also TCP/IP checksums are small enough that you'll relatively frequently see errors sneak through them. But that's not likely to be what you were noticing.

The TCP checksum is also only 2 bytes, so for large files over bad connection, corruption might slip by.

The TCP checksum is especially weak even for its tiny size. And many devices that are built to alter packets recalculate the checksum even when they're not altering anything-- so if they're corrupted in the memory of bus of these devices their checksums will get fixed. :(

Back in the Windows 98 days, I was transferring my MP3 collection over my network to my other PC. For some reason on this day, my network hub started showing massive collisions (there were only 2 computers plugged into it). I figured that if there were network errors, it would retry and grab the bad parts of the file again. Nope. Corrupted every single MP3 that I had collected. Every song had transferred, but they all ended up with loud pops and clicks and buzzing. Still not sure why that happened to this day.

I think you were referring to this part with "errors creeping in", such as the Usenet storage servers probably using cheap/old drives & controllers with bit-rot & bit-flips before it got to the TCP/IP stack.

Probably also using cheap/old network equipment too that would bit-flip between when the TCP/IP checksum was recalculated. And like you said, combined with the re-publishing of corrupted files if they didn't bother to par2 correct them first (ie., human errors creeping in :)

I very much agree that we should do better, and it's baffling that we're not.

> Surely these would be extremely useful for long-tem storage formats?

I suspect most people... well, most people just live with bit-rot, but the second best option is to use dumb storage and just stick a *.sha256 and maybe some par2 files next to the original data file on whatever storage you're using. Maybe even a detached GPG signature, if you care about that.

You can configure ZFS to use sha256 or sha512 for checksums. Sha256 is the default for datasets with deduplication.

Reply on news.ycombinator.com