Hacker News Reader

Comment by imtringued | Hacker News Reader

Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but not that good for analyzing them.

View on HN

> Of course graphing percentiles over time is much easier because they just represent a single value.

Ridgeline plots (joyplots) are severely underutilized.

I generally love joyplots but haven’t seen a great use of them showing latency distribution over time.

Do you know of any examples?

R https://cran.r-project.org/web/packages/ggridges/vignettes/i... ; d3 https://observablehq.com/@d3/ridgeline-plot ; py https://github.com/sbebo/joypy .

There are also violin plots, which are like a cross between a box plot and a ridgeline plot.

https://en.wikipedia.org/wiki/Violin_plot

Heatmaps sorta-kinda solve the problem of plotting histograms over time, except of course it's harder to detect differences in brightness than size.

I tend to use cumulative distribution functions when comparing distributions.

Histograms also tend to be quite easy to manipulate

Yes, absolutely, show the distribution whenever you can.

But consider production alerting purposes. For example, consider database user-lookup time. Use the 95th percentile, or even the maximum value to compare with an alert threshold. In many (most?) real-world cases system trouble shows up as a few lookup-time outliers. If you only look at averages, or even medians, you miss those outliers.

Histograms, too, bin values and thus show only a few values instead of the whole distribution.

In fact, the only big difference between histograms and percentiles are that they're sort of the same thing on a different axis.

Well, yeah, there's not much difference between a histogram and hundreds-of-percentiles. But there's a huge difference between a histogram and just a single percentile (whether it's p50, p95, p99, or whatever).

If you just want to know _if_ something changed then maybe one percentile is ok (though actually: not great). But if you are trying to figure out _what_ happened (or is happening!), a histogram is really important. Most of the time there are discrete behaviors/factors that are driving performance: timeouts, cache hits/misses, one overloaded host, a canary. The shape of the histogram will help you see those in a way that a single percentile can't.

Actually one of my coworkers gave a talk at Facebook Performance summit on this: https://www.youtube.com/watch?v=EG7Zhd6gLiw (Disclaimer: brief product pitch in the first minute as part of the speaker intro)

The main objection is that it's hard to get a 3D graph of the distribution over time. But it's still worth drawing some snapshots of the distribution.

Heat maps (showing histogram bin count as color/intensity instead of bar height) and scatterplots (potentially of a subset of data points) are both relatively straightforward ways to visualize distributions over time.

Reply on news.ycombinator.com