Comment by Retr0id | Hacker News Reader

Comment by Retr0id | Hacker News Reader

Parent Full thread

It would be nice if the CRCs and compression were optional features, but perversely that would increase the overall complexity of the format. Having compression makes it more useful on the web, which is why we're still using it today (most browsers do support BMP, but nobody uses it)

The fun thing about DEFLATE is that compression is actually optional, since it supports a non-compressed block type, and you can generate a valid stream as a one-liner* (with maybe a couple of extra lines to implement the adler32 checksum which is part of zlib)

The CRCs are entirely dead weight today, but in general I'd say PNG was right in the sweet-spot of simplicity versus practical utility (and yes, you could do better with a clean-sheet design today, but convincing other people to use it would be a challenge).

*Edit: OK, maybe more than a one-liner, but it's not that bad https://gist.github.com/DavidBuchanan314/7559825adcf96dcddf0...

Edit 2: Actual zlib deflate oneliner, just for fun:

  deflate=lambda d:b"\x78\x01"+b"".join(bytes([(i+0x8000)>=len(d)])+len(d[i:i+0x8000]).to_bytes(2,"little")+(len(d[i:i+0x8000])^0xffff).to_bytes(2,"little")+d[i:i+0x8000]for i in range(0,len(d),0x8000))+(((sum(d)+1)%65521)|(((len(d)+sum((len(d)-i)*c for i,c in enumerate(d)))%65521)<<16)).to_bytes(4,"big")

>The CRCs are entirely dead weight today

Why?

The usual answer is that "checksumming should be part of the FS layer".

My usual retort to such an assertion is that filesystem checksums won't save you when the data given to the FS layer is already corrupted, due to bit flips in the writer process's memory. I personally have encountered data loss due to faulty RAM (admittedly non-ECC, thanks to Intel) when copying large amounts of data from one machine to another. You need end-to-end integrity checks. Period.

I agree with the "usual" answer, or more generally, "the layer above". We shouldn't expect every file format to roll its own error detection.

If you truly care about detecting bit-flips in a writer process's memory, that's a very niche use-case - and maybe you should wrap your files in PAR2 (or even just a .zip in store mode!).

99% of in-the-wild PNGs are checksummed or cryptographically signed at a layer above the file format (e.g. as part of a signed software package, or served over SSL).

Edit: Furthermore, the PNG image data is already checksummed as part of zlib (with the slightly weaker adler32 checksum), so the second layer of checksumming is mostly redundant.

> We shouldn't expect every file format to roll its own error detection.

On the other hand, why not? If you are dealing with files that are usually 200kB+, putting 4 or 16 bytes towards a checksum is not a big deal and can help in some unusual situations. Even if the decoder ignores it for speed, the cost is very low.

The space cost is negligible, but the time cost for the encoder is real. Since most decoders do verify checksums, you can't just skip it. Take fpng[1] as an example, which tries to push the boundaries of PNG encode speed.

> The above benchmarks were made before SSE adler32/crc32 functions were added to the encoder. With 24bpp images and MSVC2022 the encoder is now around 15% faster.

I can't see the total percentage cost of checksums mentioned anywhere on the page, but we can infer that it's at least 15% of the overall CPU time, on platforms without accelerated checksum implementations.

[1] https://github.com/richgel999/fpng

I didn't infer 15% from the way it was written there. But most platforms these days have some form of CRC32 "acceleration". Adler32 is easy to compute so I'm even less concerned there.

I spent a bunch of time optimising the code in fpnge (https://github.com/veluca93/fpnge), which is often notably faster than fpng (https://github.com/nigeltao/qoir/blob/5671f584dcf84ddb71e28d...), yet checksum time is basically negligible.

Having said that, the double-checksum aspect of PNG does feel unnecessary.

Does 15% more time to encode matter? How much time is spent encoding files vs decoding? That is probably still an negligible amount of compute, out of the total compute spent on PNGs.

Your specific number seem to come from an (old version of) an encoder that has super-optimized encode and not (yet) optimized CRC.

If the 15% didn't matter, the optimization wouldn't have been made.

CRC can't save you from faulty RAM. It can save you from bitrot in data at rest and from transmission errors. If you have faulty RAM, all bets are off. The data could be corrupted after it's been processed by the CPU (to compute the CRC) and before it's been sent to the storage device.

Arguably, the real reason CRC is useless is that most people don't care about the data integrity of their PNGs. Those who do care probably already have a better system of error detection, or maybe even correction.

My retort is "please please give me a filesystem for windows that checksums and isn't highly buggy".

Edit 3: simplified the adler32 implementation

  deflate=lambda d:b"\x78\x01"+b"".join(bytes([(i+0x8000)>=len(d)])+len(d[i:i+0x8000]).to_bytes(2,"little")+(len(d[i:i+0x8000])^0xffff).to_bytes(2,"little")+d[i:i+0x8000]for i in range(0,len(d),0x8000))+(((sum(d)+1)%65521)|(((sum((len(d)-i)*c+1 for i,c in enumerate(d)))%65521)<<16)).to_bytes(4,"big")

Reply on news.ycombinator.com