Hacker News Reader

Comment by jgrahamc | Hacker News Reader

We took a very detailed look at Brotli performance (both speed and compression) and concluded that for dynamic content it was only useful if the file size was greater than 64k on slow connections and there's a very detailed description here:

https://blog.cloudflare.com/results-experimenting-brotli/

If you want to experiment with it, it's enabled on CloudFlare's test server https://http2.cloudflare.com/.

Since we don't charge by byte the focus is on end-user performance and not 'saving money' and we concluded that it wasn't currently worth implementing Brotli widely but will continue to experiment with it.

View on HN

Seems important to re-iterate that for static content (javascript and css bundles, static HTML sites), Brotli is a definitive win. I appreciated that the author was clear that the disadvantages applied to dynamic content, by always qualifying "content".

There's a lot of stuff greater then 64k nowadays ;)

Exactly :) If Cloudflare decides it’s worth several man-days to offer this feature to their clients who deliver larger files than 64k, feel free to reach out to us, we’ll be happy to share our experience :) devs@cdn77.com

Thanks. We actually wrote and open sourced our nginx modifications for choosing between Brotli and gzip.

https://github.com/cloudflare/ngx_brotli_module

From the Cloduflare's tests:

"Most files are smaller than 64KB, and if we look only at those files then Brotli 4 is actually 1.48X slower than zlib level 8!"

And the faster zlib levels (1-7) are even faster! See the table. I especially like zlib 1.

Keep a rolling average of each user's request filesize and if it exceeds 64KB (image, video, pdf, downloads) then notify them of a possible performance boost to switch to Brotli.

> if it exceeds 64KB (image, video, pdf, downloads) then notify them

All these can't be compressed more than they already are, forcing any lossless (like zlib and brotli) compressor on them is just the loss of CPU time.

Try and compare.

Compression 101.

There are actually some cases where you can gain by losslessly compressing these. For example, EXIF data in a JPEG is plain text, so gzip can help:

    $ curl -sS http://www.exiv2.org/include/img_1771.jpg | wc -c
    32764
    $ curl -sS http://www.exiv2.org/include/img_1771.jpg | gzip -9 | wc -c
    31323

In this case, though, it often makes more sense to just remove the EXIF data.

PDF files can be even more compressible, at least ones that are text and layout as opposed to embedded images:

    $ curl -sS  http://www.polyu.edu.hk/iaee/files/pdf-sample.pdf | wc -c
    7945
    $ curl -sS  http://www.polyu.edu.hk/iaee/files/pdf-sample.pdf | gzip -9 | wc -c
    4336

I agree, there is a small subset of such files where small pieces (or even the whole) can be compressed. If we just say "images" or "videos" there are also "uncompressed" variants of such, but such images or videos are always those which aren't even prepared to be on the web. E.g. "uncompressed AVI" or "Windows BMP." Those prepared for the web (e.g. MP4 or PNG) certainly aren't losslessly compressible.

Those that want to deliver the videos or pictures should use the proper format and not depend on Brotli or zlib.

Well they might compress more, but if so they should publish ;)

Gzip/brotli compression is not done for media, typically only for plain text files.

Especially those pesky XML and JSON payloads!

Couldn't you implement it for all files greater than 64K?

Yes, but that would not make sense.

CloudFlare is very close to end users (single digit milliseconds) and gzip is fast and gives good compression. What we can't afford to do is make a file smaller at the cost of increased latency because the compression time was longer. There's a trade off between compression time and delivery time.

So, Brotli makes sense for things we cache (because we can compress out of band) and for large files delivered over slow connections but is not a panacea.

You can even keep cached files in ram with brotli to save ram. Nvm you'd need 3 versions then.

You guys do some of the most interesting blog postings.

I work a block from your office in SF; may I make a request: can you guys host a public tech-talk so I can come to your office and hear it straight from your engineers and maybe have the opportunity to ask questions.

Your Writeup about the HFT NICs from solar(something) circa ~2012 has had me hooked on cloudflares opinion on pretty much everything.

The only other Corp blog post I love as much as yours are the ones from Backblaze re their pods.

That's definitely something we're planning to do more often now that we have a great space for it (both in San Francisco and London).

I love their blog but it's dead this month, I visit twice a day

Lots of great stuff going on internally that has left little time for writing. Will get back to sharing more soon. Thanks for the gentle prod.

Nice to hear

Here you go: https://blog.cloudflare.com/tls-certificate-optimization-tec...

Perfect

Why can't you only do it on cache hits?

Reply on news.ycombinator.com