For plain old zip this is excusable since it comes from the 80s, but for modern compression utilities they really should do better.
For plain old zip this is excusable since it comes from the 80s, but for modern compression utilities they really should do better.
[1]: Benchmarks available at http://facebook.github.io/zstd/#benchmarks
Which is really unfortunate, because bzip2 is not a very good compressor otherwise. It outperforms gzip on compression ratio, but is beat out by anything modern, and is extremely slow, even when parallelized.
A lot of modern compressors can be made parallelizable (and, as a bonus seekable!) by breaking the input/output into chunks and compressing each chunk independently. This loses a bit of compression efficiency, but surprisingly little for sufficiently large blocks.
(This isn't what pigz does, by the way. pigz keeps state between each compressed block, which prevents parallel decoding.)
> For plain old zip this is excusable
Do you mean gzip? PKZIP decompression is actually somewhat parallelizable, because each file is compressed independently of the other files in the archive -- I'm not sure anyone actually does this, though.
What you can't do is decompress a single archive member in parallel.
That's odd. Zip and Rar, at least, have indexes that point to individual files in the archive and it should be trivial to decompress files in a thread pool with as many cores you want. That's not as easy when you have a tar.bz or tar.gz2 file because you only see files as the .tar file is extracted.