Hacker News Reader

Comment by mastax | Hacker News Reader

My impression (not based on anything firm) is that basic reasonable-length CRCs are extremely unlikely to have collisions, unless someone is deliberately trying to create collisions. If I were looking for insurance I'd turn to one of the fast 128-bit non-cryptographic hashes.

View on HN

I used to download movies and stuff from Usenet and I noticed that there were a lot of errors in the files even though TCP/IP uses checksums. I assumed this came from people downloading and uploading the files repeatedly and errors creeping in.

That's why these days I always use BitTorrent, because it uses cryptographic hashes (albeit the outdated SHA-1) and I'm therefore guaranteed that every bit is identical to the source.

The problem isn't the checksum algorithm, it's that TCP/IP only protects packets in-flight. The checksums aren't kept on-disk, so errors can creep in after they've made it across the network.

Well, and also TCP/IP checksums are small enough that you'll relatively frequently see errors sneak through them. But that's not likely to be what you were noticing.

The TCP checksum is also only 2 bytes, so for large files over bad connection, corruption might slip by.

The TCP checksum is especially weak even for its tiny size. And many devices that are built to alter packets recalculate the checksum even when they're not altering anything-- so if they're corrupted in the memory of bus of these devices their checksums will get fixed. :(

Back in the Windows 98 days, I was transferring my MP3 collection over my network to my other PC. For some reason on this day, my network hub started showing massive collisions (there were only 2 computers plugged into it). I figured that if there were network errors, it would retry and grab the bad parts of the file again. Nope. Corrupted every single MP3 that I had collected. Every song had transferred, but they all ended up with loud pops and clicks and buzzing. Still not sure why that happened to this day.

I think you were referring to this part with "errors creeping in", such as the Usenet storage servers probably using cheap/old drives & controllers with bit-rot & bit-flips before it got to the TCP/IP stack.

Probably also using cheap/old network equipment too that would bit-flip between when the TCP/IP checksum was recalculated. And like you said, combined with the re-publishing of corrupted files if they didn't bother to par2 correct them first (ie., human errors creeping in :)

Reply on news.ycombinator.com