Hacker News Reader

Comment by ncmncm | Hacker News Reader

Zip is now effectively obsolete. Good ones lately are Zstd and Lz4, depending on your needs. Lz4 doesn't compress as much, but it is very, very fast.

Zstd compresses better than Zip, and unpacks many times faster. It can re-use a custom dictionary saved off from a previous compression, so you can get quick startup time for later fragments.

View on HN

You're not wrong, but the opening statement "Zip is now effectively obsolete." is sort of true but not really in a lot of contexts. It isn't something I'd say with conviction.

From your point of view was the implied statement something akin to: "In the context of a high-throughput network transfer program choosing DEFLATE over an algorithm more well optimized for computational weight wouldn't be a good idea." ?

Nitpicking a bit, but a format that is still used ~90% of the time is not obsolete. Not optimal (anymore) is what you meant, I think.

I mean there is no objective reason to use gzip (deflate) compression anymore, unless whoever has to use the data cannot themselves upgrade and is stuck with a gzip deflator.

It is little different from building with ARMs instead of Z80s. Where Z80s were once a good choice, now they are almost always a bad choice. Obsolete. USB sticks vs floppies, OLED vs fluorescent-backed LCD, valacyclovir vs acyclovir, Google vs Altavista, Git vs Subversion vs RCS. Need I go on?

*... with a gzip inflater.

With zstd's training mode, why is there a need for custom compression algorithms for genomics? Are they able to compress more or compress faster?

Zip is just a file format, it supports multiple compression algorithms. You can put any compression algorithm in a zip implementation. I mentioned it because it's the oldest and most well known.

The parent would probably be best off using one of the specialty genomics compression tools, that's why I linked to them.

OK, I was really comparing to zstd and lz4 to gzip, which was the original algorithm in zip, and the one most people still use. Clearly, if you can plug better algorithms into Zip, you should, providing the Zip archive format is still good enough.

I thought ZIP files only supported a bunch of old algorithms like deflate. Maybe you can make an exotic zip file with zstd but doubt there are any tools that will understand it.

Version 6.3.3 from the Library of Congress - https://www.loc.gov/preservation/digital/formats/digformatsp...

Writeup on zip file Appnotes: https://entropymine.wordpress.com/2019/08/22/survey-of-zip-a...

Actual PKWare AppNotes: https://support.pkware.com/display/PKZIP/Application+Note+Ar...

Section 4.4.5 is "compression method"

Reply on news.ycombinator.com