Hacker News Reader

Comment by rwmj | Hacker News Reader

https://github.com/facebook/zstd/issues/395#issuecomment-535...

If anyone from Facebook is reading, please look at making zstd seekable. There is already experimental support but it needs to be standardized. This could open up really interesting new use-cases [see my comment linked above], where currently we have to use much slower xz/lzma instead.

View on HN

So, xz isn't as much of a compression algorithm as a compression container format (which is a big reason why it generates slightly larger files than directly using the lzma tooling, and also the reason why a lot of people dislike it for solving a kind of weird problem); why not just add a filter type for zstd to xz? Adding this support to zstd when we already have this awkward container format to solve this specific problem in isolation seems like the wrong abstraction.

Sure - the problem isn't really on a technical level, it's the social problem of getting everyone to agree to adhere to one standard and then we can implement that in a single set of formats and tools.

You can use sqlite with zstd compression (which can be extremely efficient with a shared compression dictionary).

I'm actually working on a sqlite addon right now that allows to transparently compress tables in sqlite using zstd dictionaries and the preliminary results are really promising! One DB I have compresses down from 1GB to 80MB and random SELECTs are faster in the compressed DB when the cache is cold and only somewhat slower when it is hot :)

https://github.com/phiresky/sqlite-zstd

Reply on news.ycombinator.com