Hacker News Reader

Comment by gjvc | Hacker News Reader

The real trick is to use a parallelized implementation of the compression tool, viz:

    tar --use-compress-program /usr/bin/lbzip2 --create --file file.tar.bz2

or if you must use .gz

    tar --use-compress-program /usr/bin/pigz --create --file file.tar.gz

View on HN

Wow, that's really neat. I have never heard of pigz before despite being an avid Ubuntu user for a long time. Also, I see it supports the "-11" level which is using the zopfli algorithm!

Appreciate you sharing.

Is there a way to decompress in parallel? Afaik pigz can use a few threads to separate decompressing and i/o -- is there anything can we can pass to tar to ungzip faster?

Not without changing the structure of the gzip unfortunately.

Gzip files are allowed to consist of multiple streams. Bgzip (block gzip) is one tool that reads and writes gzip-compatible files in parallel. The output can be decompressed in parallel by bgzip, or semi-random access, or decompressed normally by gzip.

Lzip can be decompressed in parallel if compressed with "tarlz".

https://www.nongnu.org/lzip/tarlz.html

pzstd can decompress in parallel if the data was compressed using pzstd. Although zstd is so much faster for decompression that you rarely need multiple threads.

For xz, pixz can decompress in parallel.

I have been working on exactly that: pragzip. It decompresses in parallel in a similar manner to the prototype pugz by implementing a two-staged decompression. https://github.com/mxmlnkn/pragzip I also did a Show HN: https://news.ycombinator.com/item?id=32366959

As a FYI pigz also can speed up decompression as well, but the performance boost is minimal. The decompression is still in a single thread but it spawns other threads for i/o and so forth. For compression the speedup is quite linear.

igzip (https://github.com/intel/isa-l) is much faster than gzip or pigz when it comes to decompression, 2-3x in my experience. There is also a Python module (isal) that provides a GzipFile-like wrapper class, for an easy speed-up of Python scripts that read gzipped files.

However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip. You also need to make sure to use the latest version if you are going to use it in the context of bioinformatics, since older versions choke on concatenated gzip files common in that field.

> However, it only supports up to level 3 when compressing data, so it can't be used as a drop-in replacement for gzip.

[EDIT: I misread a benchmark result [1] and wrongly concluded that igzip -3 is better than gzip -9, in reality it is somewhere in the middle of -1 and -9.]

[1] https://encode.su/threads/3398-Sequence-Compression-Benchmar...

That benchmark seems to be comparing igzip with gzip -1, since the reported gzip output size is 42298786 bytes and not 36548933 bytes (from the earlier benchmarks). My own experience based mostly on FASTQ/text data is that igzip -3 gives results very similar to gzip -3.

Oops! Yes, you are correct. I confused it with an other reply with level 9 benchmarks, please disregard the parent comment.

lrzip

Reply on news.ycombinator.com