Hacker News Reader

Comment by ed25519FUUU | Hacker News Reader

Looks great! I’m still surprised these tools (gzip, etc) haven’t been update to take advantage of multicore support.

In the gzip case, the reason is that the compressed stream format is inherently sequential: you need to decompress everything before a byte before you can decompress the next byte. So parallizing is not really possible, without changing the format.

Newer compression formats have multiple streams that can be decompressed in parallel, or have blocks that are compressed independently without dependencies (bzip2 can be decompressed in parallel due to independent blocks).

So is Pigz not backward compatible with gzip?

I don't think Pigz decompresses in parallel.

Pigz is backward compatible — it breaks the input into 128K chunks, compresses each in parallel, and concatenates them. Gzip supports naive concatenation of compressed files so the result can be decompressed by standard gzip.

Pigz does not implement parallel decompression, but pigz-compressed files could, in principle, be decompressed in parallel. I assume there's not much urgency for this because gzip decompression is fast enough: >3x faster than compression and >10x faster than bzip2 decompression on default settings. (See: https://quixdb.github.io/squash-benchmark/ )

It's good that they haven't. If you're compressing a large number of individual files (rather than combining them into a single archive), it's still more efficient to run multiple copies of gzip or whatever via something like "parallel".

Good point. I don't think anything bad would happen if you could opt in via something like `-j8`, preserving the default backwards-compatible behaviour and offering an escape hatch for newer workloads / machines.

Reply on news.ycombinator.com