But the problem with averaging averages is that it makes no sense mathematically. The average (well, arithmetic mean, to be technical) has a well-defined meaning and interpretation in terms of the original data points. When you start averaging averages (or dear Lord percentiles!) you lose the last glimmer of interoperability you had.
The bigger issue here, aside from the fact that you may be taking a simple average when you need to be taking a weighted average to aggregate your individual point estimates, as others have pointed out, is that you can estimate a sufficiently symmetric Poisson distribution with a normal distribution, but response latency is not symmetric, specifically because time travel isn't possible and you can't ever get negative response times to balance out the tail events on the right side of the distribution. Tail events with respect to latency can only be bad, not good. They're a bet with a hard cap on winnings but potentially unbounded losses. As such, you really need to consider what the worst case is and the expected frequency of that worst case.
For one great deep dive on the topic, check out https://drkp.net/papers/latency-socc14.pdf
I suspect, naively, that latency purely from network appliance queueing probably gives a mixture of Poissons in a straight line bus topology. But add in arbitrary routing and the much more complicated schedulers of the servers themselves, and you get no well-behaved distribution at all.
It's especially interesting to compare the SLAs given by data center providers that tend to always use 99th or even 99.9th percentile worst case performance, with application metrics using arithmetic averages.
Human height?
- Minute 1: 10 requests, 5 with 10ms latency, 5 with 500ms latency; avg latency 255ms
- Minute 2: 1000 requests, 900 with 10ms latency, 100 with 500ms latency; avg latency 59ms
Average of averages: 157ms
Average latency: 97ms
You can make them arbitrarily different.
> (255ms + 59ms) / 2 = 157ms
But
> ((10 * 255ms) + (1000 * 59ms)) / 1010 = 60.94ms
Which is the same answer I get as averaging all the samples without bucketing. (I can't get 97ms though)
So if your response is, “can’t you just collect the correct data in the first place?” Well, that’s the hard part.
If all you have is 50 averages, there is no way to combine them to get the average of the underlying data, because you don't have the original "length(x)".