Hacker News Reader

Comment by jl6 | Hacker News Reader

The format is fine, and zstd would be a downgrade in terms of compression ratio (which is the main reason xz is chosen for its current use cases). The lesson to learn here is not about making technical improvements, it’s to do something about improving the way we manage and maintain core open source infrastructure.

Ref XKCD 2347.

View on HN

No, xz was chosen mainly because zstandard didn't exist at that time [1]. Zstandard does almost completely replace those "expensive" compression formats by providing a comparable compression ratio but being much faster.

[1] https://lists.debian.org/debian-devel-announce/2011/08/msg00...

"almost" and "comparable", but xz (lzma) still compresses at a higher ratio, which is still a highly desirable feature.

If the decision was being made today, a good faith debate could be had over whether the distro packaging use case benefits more from faster decompression or smaller file size, but I suspect that, globally, the average PC can decompress xz faster than the average broadband connection can download it.

I will assume that the average effective broadband connection is about 20 MB/s, and both xz and zstd decompressions are faster than the download under this assumption. But there are lesser known compression algorithms that are more efficient in terms of compression ratio for this target decompression speed, like zpaq [1], which are rarely considered as alternatives. So multiple factors exist here and I believe zstd excels at pretty much every other factor compared to xz, making it a very likely choice.

[1] https://mattmahoney.net/dc/zpaq.html

It's not, zstd can match LZMA2 compression ratio, while being much faster.

Much faster: yes.

While matching LZMA2 compression ratio: no.

The format is fine, except for https://www.nongnu.org/lzip/xz_inadequate.html

Yeah, that title is click bait. The format is perfectly fine for long term archival.

In fact, I'd wager that all the focus that article has put on the minor problems with the format has taken much needed attention away from the fact that the maintainership makes xz-utils inadequate for long-term archival...

That attack article puts forward a dubious scenario where parts of multiple copies of a file become corrupted in different locations and you need to recover the original using both of the damaged copies.

If you find that sort of thing happening to you a lot, then lzip might be the format for you. But I can’t say I’ve ever heard of that being a real recovery scenario.

A much saner approach to enabling recovery from corruption would be to keep external checksums and parity files.

Reply on news.ycombinator.com