Comment by saagarjha | Hacker News Reader

Comment by saagarjha | Hacker News Reader

Parent Full thread

> This also applies to compression and decompression performed by Archive Utility: for example, if you download a copy of Xcode in xip format, decompressing that takes a long time as the code is constrained to the E cores, and there’s no way to change that.

This is actually not correct. If you decompress something with Archive Utility, you'll see that it does spend some time on all the cores, although it's definitely not using them equally. It's just that it doesn't spend very much time on them, because it does not parallelize tasks very well, so most of the time it will be effectively running on 1-2 threads with the other cores idle, which macOS will split over the efficiency cores and perhaps bring in one of the P-core clusters if it likes. For the very limited portions of the process that Archive Utility is actually computationally bottlenecked, AppleFSCompression will farm out the work to all the cores. When writing unxip (https://github.com/saagarjha/unxip) I found that you will actually lose to Archive Utility if you don't use all the cores at least part of the way.

> There are two situations in which code appears to run exclusively on a single P core, though: during the boot process, before the kernel initialises and runs the other cores, code runs on just a single active P core.

I'm not 100% sure on this, but I believe the processor comes out of reset on an Icestorm core?

It's very annoying that of all of the commonly used compression formats in use today it seems like only BZip2 parallelizes nicely. Some will use all of the cores on compression but only one for decompression and others do all of the math on a single core.

For plain old zip this is excusable since it comes from the 80s, but for modern compression utilities they really should do better.

I believe zstd does parallel compression/decompression and has some of the best efficiency ratios.[1] I’ve been replacing gzip with zstd as a general purpose compression format.

[1]: Benchmarks available at http://facebook.github.io/zstd/#benchmarks

So, zstd is gzip replacement? lower compression ratio than bzip2 and much much faster. Also better in both compression and speed than gzip

With the right options, zstd can compress quite a bit better than bzip2 and gzip. Albeit probably not as fast as gzip, because squeezing out that extra compression space is a trade for speed.

> it seems like only BZip2 parallelizes nicely

Which is really unfortunate, because bzip2 is not a very good compressor otherwise. It outperforms gzip on compression ratio, but is beat out by anything modern, and is extremely slow, even when parallelized.

A lot of modern compressors can be made parallelizable (and, as a bonus seekable!) by breaking the input/output into chunks and compressing each chunk independently. This loses a bit of compression efficiency, but surprisingly little for sufficiently large blocks.

(This isn't what pigz does, by the way. pigz keeps state between each compressed block, which prevents parallel decoding.)

> For plain old zip this is excusable

Do you mean gzip? PKZIP decompression is actually somewhat parallelizable, because each file is compressed independently of the other files in the archive -- I'm not sure anyone actually does this, though.

Last time I looked the information I found online said that due to the way PKZIP files are structured it isn't possible to decompress them in parallel. But checking again it seems that someone had done just that:

https://github.com/jpakkane/parzip

That information was incorrect. Any member in a PKZIP archive can be decompressed independently of any other files in the archive, which is what parzip does.

What you can't do is decompress a single archive member in parallel.

The Xcode XIP is chunked and can be decoded in parallel, but not seekable unfortunately.

Switching from gzip->pigz really sped things up in one of our CI steps recently. It seemed like the output files were identical as well. What are the drawbacks?

> it seems like only BZip2 parallelizes nicely.

That's odd. Zip and Rar, at least, have indexes that point to individual files in the archive and it should be trivial to decompress files in a thread pool with as many cores you want. That's not as easy when you have a tar.bz or tar.gz2 file because you only see files as the .tar file is extracted.

Easy to say, hard to do. Particularly since most threading optimizations will trade some bandwidth - and people generally prefer smaller file sizes as their first-order preference.

The files are typically within a few bytes, there's really no practical difference in size.

Reply on news.ycombinator.com