Hacker News Reader

Comment by ars | Hacker News Reader

zstd hardly replaces xz, the compression ratio is quite worse. zstd seem more of a replacement for gz.

Most tests I've seen, such as [0] don't support your statement. Zstd can compress almost as well as xz but decompresses much faster.

It can also compress more than xz with tweaks, though I don't know the compute/memory tradeoffs.

[0] https://archlinux.org/news/now-using-zstandard-instead-of-xz...

Edit to fix typo.

Sure, if you look at the Pareto frontier for xz and zstd, zstd does not seem like a “replacement” for xz. It’s not a replacement for PPMd.

The problem is that xz has kind of horrible performance when you crank it up to the high settings. On the medium settings, you can get the same ratio for much, much less CPU (round-trip) by switching to zstd.

YMMV, use your own corpus and CPU to test it.

Depends on your use case. Saving the last couple of Bytes is often less important than fast compression.

Do you have any data to support this claim? In my experience, zstd is way better in every way compared to gzip. Additionally, xz is good compression but horribly crazy slow to decompress. Xz also only operates on one single file at a time, which is annoying.

I just did a quick test with a copy of GNU M4, which is reasonably representative of a source code archive.

  $ time xz -9k m4-1.4.19.tar 
  real 0m2.928s
  user 0m2.871s
  sys 0m0.056s
  $ time zstd -19 m4-1.4.19.tar
  real 0m3.411s
  user 0m3.380s
  sys 0m0.032s
  $ ls -l m4-1.4.19.tar*
  -rw-rw-r-- 1 john john 14837760 Jul 24 14:40 m4-1.4.19.tar
  -rw-rw-r-- 1 john john  1674612 Jul 24 14:40 m4-1.4.19.tar.xz
  -rw-rw-r-- 1 john john  1726155 Jul 24 14:40 m4-1.4.19.tar.zst

In this test, XZ was both faster and had better compression than Zstd.

Howdy jmillikin.

That is a very small file.

See: https://news.ycombinator.com/item?id=25455846#:~:text=xz%20o....

> xz on highest (normal) compression level easily beats the compression ratio of zstd on highest compression level (--ultra -22) on any data I've tested. However with xz reading the compressed files easily becomes a bottleneck, zstd has great read speeds regardless of compression ratio

Are you agreeing or disagreeing with ars' claim that XZ provides a better compression ratio than Zstd? My data shows that it's true in at least one common use case (distribution of open-source software source archives).

I've seen similar comparative ratios from files up to the multi-gigabyte range, for example VM images. In what cases have you seen XZ produce worse compression ratios than Zstd?

Generally speaking, the top-end of xz very slightly beats the top-end of zstd. However, xz typically takes several times as long to extract. And generally I've seen xz take longer to compress than zstd, as well.

Example with a large archive (representative of compiled software distribution, such as package management formats):

    $ time xz -T0 -9k usrbin.tar 
    
    real 2m0.579s
    user 8m46.646s
    sys 0m2.104s
    
    $ time zstd -T0 -19 --long usrbin.tar 
    real 1m47.242s
    user 6m34.845s
    sys 0m0.544s
    /tmp$ ls -l usrbin.tar*
    -rw-r--r-- 1 josh josh 998830080 Jul 23 23:55 usrbin.tar
    -rw-r--r-- 1 josh josh 189633464 Jul 23 23:55 usrbin.tar.xz
    -rw-r--r-- 1 josh josh 203107989 Jul 23 23:55 usrbin.tar.zst
    /tmp$ time xzcat usrbin.tar.xz >/dev/null

    real 0m9.410s
    user 0m9.339s
    sys 0m0.060s
    /tmp$ time zstdcat usrbin.tar.zst >/dev/null
    
    real 0m0.996s
    user 0m0.894s
    sys 0m0.065s

Comparable compression ratio, faster to compress, 10x faster to decompress.

And if you do need a smaller compression ratio than xz, you can get that at a cost in time:

    $ time zstd -T0 -22 --ultra --long usrbin.tar 

    real 4m32.056s
    user 9m2.484s
    sys 0m0.644s
    $ ls -l usrbin.tar*
    -rw-r--r-- 1 josh josh 998830080 Jul 23 23:55 usrbin.tar
    -rw-r--r-- 1 josh josh 189633464 Jul 23 23:55 usrbin.tar.xz
    -rw-r--r-- 1 josh josh 186113543 Jul 23 23:55 usrbin.tar.zst

And it still takes the same amount of time to extract, 10x faster than xz.

That seems fine -- it's a tradeoff between speed and compression ratio, which has existed ever since compression went beyond RLE.

Zstd competes against Snappy and LZ4 in the market of transmission-time compression. You use it for things like RPC sessions, where the data is being created on-the-fly, compressed for bandwidth savings, then decompressed+parsed on the other side. And in this domain, Zstd is pretty clearly the stand-out winner.

When it comes to archival, the wall-clock performance is less important. Doubling the compress/decompress time for a 5% improvement in compression ratio is an attractive option, and high-compression XZ is in many cases faster than high-compression Zstd even delivering better ratios.

---

EDIT for parent post adding numbers: I spot-tested running zstd with `-22 --ultra` on files in my archive of source tarballs, and wasn't able to find cases where it outperformed `xz -9`.

I think you're missing the point that in terms of tradeoffs people are willing to make: absolute compression ratio loses to 80% of the compression ability with big gains to decompression speed (aka include round trip cpu time if you want something to agree / disagree with, we're not talking about straight compression ratios).

Arch Linux is a case study in a large distributor of open source software that switched from xz compressed binaries to zstd and they didn't do it for teh lulz[0].

[0] https://archlinux.org/news/now-using-zstandard-instead-of-xz...

I'm not missing the point. I'm responding to the thread, which is about whether XZ offers better compression ratios than Zstd.

Whether it's faster in some, many, or most cases isn't really relevant.

Yup and how much better is about 1%. "zstd and xz trade blows in their compression ratio. Recompressing all packages to zstd with our options yields a total ~0.8% increase in package size on all of our packages combined, but the decompression time for all packages saw a ~1300% speedup."

Yes, because ratio is the only thing that matters. Decompression speed doesn't matter at all. Who cares if Zstd is 9 times faster?

Reply on news.ycombinator.com