Just one example that comes to mind: suppose you want average (mean) latency. So you record, for every minute, the average latency for requests during that minute. Now you want to calculate the average latency for an entire day… but it’s impossible. You can calculate the daily average of the per-minute averages, but you’re averaging over minutes, rather than averaging over requests.
This is just a simple example, but you can see how a seemingly innocuous decision sabotaged your ability to run the query you want.
Maybe my advice is “learn calculus”.
So, well meaning engineers will collect percentiles on individual components in the system, with the idea that if you want percentiles at the end, just collect percentiles everywhere. You’re left with garbage data that can’t be aggregated in a way that makes any sense.
Histograms can be added and aggregated. You can always estimate the mean from a histogram.
So if you compute a histogram of latency values every minute, you can add them up for successive minutes to get the distribution over larger intervals.
But the problem with averaging averages is that it makes no sense mathematically. The average (well, arithmetic mean, to be technical) has a well-defined meaning and interpretation in terms of the original data points. When you start averaging averages (or dear Lord percentiles!) you lose the last glimmer of interoperability you had.
The bigger issue here, aside from the fact that you may be taking a simple average when you need to be taking a weighted average to aggregate your individual point estimates, as others have pointed out, is that you can estimate a sufficiently symmetric Poisson distribution with a normal distribution, but response latency is not symmetric, specifically because time travel isn't possible and you can't ever get negative response times to balance out the tail events on the right side of the distribution. Tail events with respect to latency can only be bad, not good. They're a bet with a hard cap on winnings but potentially unbounded losses. As such, you really need to consider what the worst case is and the expected frequency of that worst case.
For one great deep dive on the topic, check out https://drkp.net/papers/latency-socc14.pdf
I suspect, naively, that latency purely from network appliance queueing probably gives a mixture of Poissons in a straight line bus topology. But add in arbitrary routing and the much more complicated schedulers of the servers themselves, and you get no well-behaved distribution at all.
It's especially interesting to compare the SLAs given by data center providers that tend to always use 99th or even 99.9th percentile worst case performance, with application metrics using arithmetic averages.
Human height?
- Minute 1: 10 requests, 5 with 10ms latency, 5 with 500ms latency; avg latency 255ms
- Minute 2: 1000 requests, 900 with 10ms latency, 100 with 500ms latency; avg latency 59ms
Average of averages: 157ms
Average latency: 97ms
You can make them arbitrarily different.
> (255ms + 59ms) / 2 = 157ms
But
> ((10 * 255ms) + (1000 * 59ms)) / 1010 = 60.94ms
Which is the same answer I get as averaging all the samples without bucketing. (I can't get 97ms though)
So if your response is, “can’t you just collect the correct data in the first place?” Well, that’s the hard part.
If all you have is 50 averages, there is no way to combine them to get the average of the underlying data, because you don't have the original "length(x)".
To go to the next level I would suggest also emitting the sum and sum of squares, which will make your summary statistic far more useful.
And choose the right quantization for histograms/cdfs.
> To go to the next level I would suggest also emitting the sum and sum of squares, which will make your summary statistic far more useful.
Histograms should also give you the sum. I haven't seen sum of squares in many monitoring setups; is this for underlying metrics that are differences/errors?
This allows aggregation for efficiency, usually either by partitioning the data by short time windows (1 second or more usually) and storing measures over each duration or by storing a sampled subset of original values, both of which can later be queried to produce measures over arbitrary periods of time.
A few measures can be supported exactly with pre-aggregation (min, max, mean, variance), some can be supported with bounded error (quantiles) or statistical error (pretty much any measure over sampled values).