Hacker News Reader

Comment by hinkley | Hacker News Reader

The informatics dysfunction on this graph is just off the charts.

Here's the problem. The graph is designed to make your conclusion sound right, but it doesn't actually prove that.

Let's look at what the numbers really say:

For the sample data, and the best case, gzip gets you a file that's about 31% of the original size. zstandard can get you a file that is 25% of the original size but it will take you four times as long to get it. If you allow it the same time as gzip, you can get 27% compression instead of 31%. That's only 13% improvement on the wire.

That's nice, but it's not impressive at all. It's not a good enough reason to change your stack. The only thing that is impressive is that if you want the same compression ratio as gzip you can do it up to 20 times faster. On some hardware that's totally worth it, but not on all (because who is streaming at 700 MBps?)

View on HN

Hi. I made the graph. It was definitely not my intent to obscure or distort any information. Here's the raw data if you like [1]. Some context: the input text is silesia.tar, which is a standard mixed corpus of data. It was benchmarked by @Cyan4973 on his i7-9700k.

I guess I'm confused about how you feel we're misrepresenting the data? Are you talking about the sentence preceding that graph, "The benefits we’ve found typically range from a 30 percent better ratio to 3x better speed"? That's more describing the benefits we've seen in the real world, where for a variety of reasons (many of which are discussed in the rest of the post), we generally can get more out of zstd than this vanilla benchmark shows.

[1] https://gist.github.com/felixhandte/f6a91bf775d6e7df76ce06c0...

Edward Tufte will set you straight.

The first rule of objective graphs is always show the origin at 0. As an informed graph consumer, if the origin is not 0 you should immediately distrust your eyes, and question the motives of the person showing you the data. The relative sizes are being distorted. Why?

In this case, there's a 15% difference in Y values in your data that on the graph is presented as two lines that are separated from each other by 33%. Just by removing 0 and 1 from the chart.

On the other hand, that effect is diluted a bit by what you've done on the X axis. Log scale tells a story about trends. The relative slope of two lines on log scale says something. The distance between them doesn't mean much, if anything, although the brain can't help trying to make it mean something.

I believe you will find that a great deal of what you lose by plotting y = 0 you'll gain back by plotting x on a linear scale. Those lines deserve to be much farther apart.

The Visual Display of Quantitative Information is one of my favorite books!

0 in this case is not really a relevant value (since that would mean transforming the input into something infinitely large). The functional identity value / origin here is 1x. Here's what that looks like [1].

To me, this is a significantly less useful image. But maybe that stems from a lot of comfort with both the subject matter and the detail log-log plots that I work with to evaluate zstd performance, e.g. [2].

[1] https://imgur.com/gU2Gdf6 [2] https://github.com/facebook/zstd/pull/1317#issuecomment-4260...

Hey thanks for the new chart, and you're right about 1x. But we might have to disagree because I don't think this graph is that bad. I can see a bunch of talking points in this graph that are about strategy instead of about why I hate bad graphs.

First, that data point for lz4 out around 820 MB/s kind of throws the groove off of that graph. Despite that, I would still suggest you prune the graph off at 850 or 900 to reduce the squish of the horizontal data (pruning the graph to put the last value at 100% of width or height isn't considered a no-no).

One of the things I can see now but couldn't before is that the size/speed tradeoff is pretty linear except for the giant dog-leg around 3.6:1, and a less pronounced but still notable one occurs at 2.9:1.

If I sat down with my team to discuss this chart I'd suggest we agree that we aren't interested in anything above 3.6:1. And then I'd suggest we look at everything above 2.9:1, but my eye is on 3.2:1 (where zlib tops out and is 1/10th the speed). If they don't like 2.9:1, then the next interesting point is at about 2.7:1 when zlib bottoms out. We're already at such a high bandwidth rate that something else is probably going to be the bottleneck.

Having spent the last few years in the 'real' (ie. non-software) world, it boggles me how anything less than an order of magnitude improvement is handwaved as trivial. 13% of bandwidth or 20% of offline storage space is huge savings for a company like FB. Once the software development industry matures a little, more people will start realising this.

I don't find myself disagreeing with you that much here, but 13% is not quite enough to get me to push my team to make such a big change.

On the other hand, I've also had to push for getting any compression turned on at all. If all of my infrastructure already supported zstandard, getting a 3.5:1 compression ratio would be more compelling than 3.2:1, and I can imagine situations where it's easier to get buy-in.

But first every old cell phone and crappy web browser (read: 6 year old Microsoft browser) has to support it. Which is why you have people creating backward-compatible compressors that spit out files that zlib can decompress. Because otherwise you have to support and test 3 transport formats instead of 2.

There's so much that _feels_ misleading here.

Paragraph before the chart:

"The benefits we’ve found typically range from a 30 percent better ratio to 3x better speed."

In what cases? What methodology was used to evaluate this? I certainly expected a scientific (honest) treatment of how the performance was evaluated and the corresponding trade-offs to follow on at some point.

If you read the full article there are many examples of real systems being migrated with their associated wins.

I will say that we've been 'doing just fine' with zlib for longer than I'm comfortable with. I hung out on comp.compression when I was still in college and shared the dream of writing the next great compression library. Maybe the only thing I ever accomplished though was altering a Java minifier to improve compressibility of the class files.

On a very deep level I'm pretty disappointed that zlib has been 'good enough' for almost 30 years. When it was Google proposing a change, I wasn't enthused about handing more control over HTTP to Google. They have too much already. The ways I'm concerned about Facebook have nothing to do with standards bodies or protocols. So maybe this is good enough.

There are more options for files-in-motion and files-at-rest, and that could put it over the top. But you need to be filing high quality PRs on both HAProxy and Nginx if you want anybody to care.

In short, we work with individual teams and projects to evaluate their priorities, and select or build the compression scheme that best addresses their needs. Some of those cases are described in the post, but we wanted to summarize the general kinds of results we see.

I did spend a fair amount of time working on a graphic for the post to try to capture the distribution of improvements we've seen across different use cases at Facebook, but it ended up being very hard to interpret / glean anything meaningful from.

It's difficult to present rigorous conclusions: the reality of data compression is that every use case is different, with different priorities and data characteristics that cause different compressors to behave differently.

Some data (random noise) is totally incompressible, and zstd will do just as poorly as any other algorithm. On the other hand, with highly structured / repetitive data like JSON, zstd can do enormously better than zlib. It all depends on the specific context.

Another example of how benchmarking is hard: we recently spent a fair amount of time improving zstd for highly-contended memory scenarios [1]. This work basically won't show up in a standard single-threaded single-workload benchmark. But we saw meaningful improvements in the real world at Facebook.

Ultimately, if you are evaluating using zstd (or any other compressor), the best predictor for performance will be a benchmark you run yourself with your own data. And hopefully you like what you see!

[1] https://github.com/facebook/zstd/releases/tag/v1.3.6

Reply on news.ycombinator.com