Hacker News Reader

Comment by retrac | Hacker News Reader

For that type of structured data (logs and such), a custom dictionary can be extremely effective. Zstd among others support generating a custom dictionary. You just run zstd --train over the data first, and then feed that in when you run zstd. For example: I found ~10 gigabytes of Usenet headers compress to ~700 MB using Zstd and 1 MB shared dictionary -- and that's with each header individually compressed, so o(1) lookup time.

View on HN

Back in my data hoarder days, I downloaded one of those torrents that had all the world's books in it. It was dunno how many terabytes, but way more than I had HDDs.

So I stripped out formatting, got rid of dupes, and tried out zstd, which was the hot new thing, along with the dictionary feature you describe, figuring it'd help. It didn't. I tried having one per book, one per multiple books, one for the whole archive.

It didn't work, or the gains were so marginal that I ended up scrapping the approach.

So it's not impossible that it can work, but stuff like regular json already compresses extremely well, I haven't found a scenario where it's a major boon.

I have an anecdote about compressed data.

When I studied at school, I used ZFS with lz4 enabled on my working machine. During that times I had a task of parsing Wikipedia's data. I had enough brain cells to find compressed dumps and download them with aria2 but not enough to leave the file compressed. I ran a decompressor. It'd been taking longer than I expected so I went out to walk a dog.

Imagine how fast me and the dog ran back 30 minutes later when I realized how cooked I was. I only had 10 GB left on my disks after I downloaded that 20 GB file. This decompressed file would have blown the machine up. I was terrified to find a frozen system with no storage space left.

Instead, the process finished and `df -h` reported 8 GB of the free space left. Files were decompressed. I could `less` them! That made no sense! Only many many minutes later I finally figured out to run a `zfs get compressratio` command which showed ZFS successfully and transparently recompressed everything on the fly. That was too impressive for that teenager and he never switched to a different file system.

Solaris has had so many cool features, like ZFS or doors. What I liked about ZFS is you coul make snapshot, which is basically the solution to how to treat data files a single, cheap to access unit, yet still use standard apis for file management, great for containerizing apps, making copies for experiments, or shipping stuff. Node.js just received this as a bespoke, app-level feature. But these things are too many to count, and make a ton of sense if you know how filesystems actually work. useful Also copy-on-write, temp overlays.

Sun was a really cool company.

\[T]/

as oxide is nowadays ;-)

JSONL files are likely to have a lot of the same words repeated MANY times. Same as with headers...

Because JSON is an inefficient text encoding, compression (with custom dictionary) are likely to really well on those.

Books have recurring words, but probably much less.

"Dictionary gains are mostly effective in the first few KB."

https://facebook.github.io/zstd/index.html

Pretrained dictionaries have never been intended to help with book sized or bigger compression. zstd automatically learns the most efficient dictionary it can within a few kilobytes. Pretrained dictionaries are only useful when you're independently compressing very small records.

Note that the dictionary options are only needed to improve compression ratios when compressing lots of small messages. If you have a bigger file (eg a tar file of Usenet messages) the regular Zstd compression will build a good dictionary without additional options.

As I understand there is no advantage in using a custom dictionary to compress 1 file. It benefits compressing _several_ (small) files.

Reply on news.ycombinator.com