Comment by twotwotwo | Hacker News Reader

Comment by twotwotwo | Hacker News Reader

Parent Full thread

I'd love this added as an HTTP Content-Encoding, like brotli recently was (and gzip has been for a long time). You could get a bit more compression at gzip-like speeds, or have very cheap compression using its fastest modes (hundreds of MB/s/core; see --fast modes at https://github.com/facebook/zstd/releases/tag/v1.3.4). brotli especially helped at the high end (static stuff you'll serve many times and can spend a long time compressing); zstd can be especially helpful nearer the low end (dynamic content you'll serve a single time).

On Twitter a designer of Brotli raised the question of whether there should be a standardized limit on RAM required to decode in the HTTP Content-Encoding context, since zstd supports huge window sizes (GBs). I don't feel strongly--clients can refuse to handle responses that take too much RAM to decode, servers have an incentive to serve up things their clients can handle, and there tend to be sharply diminishing returns to increasing window size--but I can imagine a limit in the MBs being a reasonable compromise between allowing high compression and not breaking on phones with a couple hundred MB RAM.

The RFC touches on that issue:

> In order to protect the decoder from unreasonable memory requirements, a decoder is allowed to reject a compressed frame that requests a memory size beyond the decoder's authorized range.

> For broader compatibility, decoders are recommended to support memory sizes of at least 8 MB. This is only a recommendation; each decoder is free to support higher or lower limits, depending on local limitations.

As a corollary, and in the absence of an explicit negotiation process between client and server, servers should probably avoid using a window larger than 8 MB.

Ah, great that there's an official number. And does seem like incentives are aligned for servers to try to do the reasonable thing.

Looks like zstd has been added to the HTTP Content Coding Registry, see https://www.iana.org/assignments/http-parameters/http-parame...

(or section 6.2 of the linked RFC)

Oh, that's great. Then a matter of browser support I guess!

Mozilla has a bug here (about Transfer-Encoding): https://bugzilla.mozilla.org/show_bug.cgi?id=1477516

Couldn't find any mentioning zstd in the public Chrome, WebKit, or MS Edge issue trackers.

Yeah, we're just getting the ball rolling. I recently added support to HHVM[1]. Pushing for community adoption generally is definitely on our agenda, but will take some time.

[1] https://github.com/facebook/hhvm/commit/d830ac5b335cea8433ab...

It looks like this was filed already: https://www.iana.org/assignments/media-types/application/zst...

As best I can see that assigns a Content-Type ("MIME type"), distinct from a Content-Encoding supported by browsers.

So it specifies what Content-Type header your browser should use for a .zst file, but doesn't represent a plan to let your HTML, JS, CSS etc. be delivered compressed using Zstd and transparently decompressed by the browser, as is possible with the gzip and br Content-Encodings.

(But turns out pauldino's comment points out it is in the "content coding registry" at https://www.iana.org/assignments/http-parameters/http-parame...)

For -de facto- unlimited dictionary size data compression (e.g. PB's using a 64-bit compiler, if you have enough memory):

git clone https://github.com/faragon/libsrt

make

./enc -ezh <hugefile >output.zh

For decompress:

./enc -dz <output.zh >hugefile.again

That's different from what I'm trying to talk about, which is how zstd might be useful as an HTTP Content-Encoding, and how you might (or might not) limit the window sizes allowed in that context to ensure usability on low-memory clients.

For small memory footprint per connection I like Willy Taureau's (HAProxy) approach in libslz: https://github.com/haproxy-unofficial-obsolete-mirrors/libsl...

SLZ is cool, but the specific question jyzg raised and my comment is talking about is whether or not any standards should spell out something like "don't use enormous window sizes with the [still hypothetical] zstd Content-Encoding, to avoid causing problems for small-memory clients decompressing your responses." This comes up because zstd (unlike gzip) lets you set your window to be GBs if you want (look for "long-range match finder" in https://github.com/facebook/zstd/releases if you want more context); those window sizes would, of course, not be ideal on a low-end smartphone, for example.

Wouldn't a 2GB window only consume that much memory if the stream is at least 2GB in size?

I bet some unpackers grab physical memory for their sliding window up front, but the balance between memory per connection and ratio, and the question of whether any standard here ought to specify a number or we should just let servers and clients do their best to work things out, would still be a thing even if the space taken were also bounded by the download's size or stream's length.

Reply on news.ycombinator.com