Hacker News Reader

Comment by Twirrim | Hacker News Reader

pigz is extremely fast and very capable, plus it's packed up and provided for almost every mainstream linux distribution, and can act as a drop in replacement for gzip as it supports the same flag syntax.

On the bzip2 side there is pbzip2, which is also a drop in replacement, http://compression.ca/pbzip2/

View on HN

pbzip2 only does parallel compression. lbzip2 can parallelize compression and decompression.

With both pbzip2 and lbzip2 I got errors every now and then while everything worked with bzip2. YMMV.

Also note that bzip2 is block based (due to bwt) and thus does not compromise compression ratios like parallelized gzip implantations.

At 32 cores you can saturate Gbit links (even with good compression ratios!). I hope we will see some more bzip2 love in the future due to it's perfect fit for parallelism.

Errors sound deeply concerning. What kind of errors? Is it too risky to use [pl]bzip2 for archival reasons?

Well, errors at a rate of one every few terabyte, and most of the time it was mainly decompression, and always caused program abort iirc.

But those have been a few years ago, so my best advice would be: retest.

You are testing your archives anyway, right? ;-)

>You are testing your archives anyway, right? ;-)

Umm no, I handle errors which I expect to be reported to me.

pbzip2 produces a separate bzip2 stream for every compressed block, which can't be read properly by many popular decoders, usually ones that are based around the bzip2 high-level interface [0]. There's a warning about this in Python 2's documentation, which is a better offering than most [1].

Many decoders simply stop reading after decompressing the first 900k without so much as a peep that something that may astonish you has happened.

  [0] http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html#hl-interface  
  [1] https://docs.python.org/2/library/bz2.html#bz2.BZ2File

There is also pixz for Xzip format. Which can be a godsend since xzip is such a slow compression. Just make sure you have the CPU to spare since it's a hog.

Xz does this natively now, with the -T parameter.

There is also pbgzip if you are into indexed access (genome data).

Reply on news.ycombinator.com