Hacker News Reader

Comment by querez | Hacker News Reader

> The geometric mean of the timings improved from 218 to 12, a ca. 20× improvement.

Why do they use the geometric mean to average execution times?

It's a way of saying twice as fast and twice as slow have equal effect on opposite sides. If your baseline is 10 seconds, one benchmark takes 5 seconds, and another one takes 20 seconds then the geometric mean gives you 10 seconds as the result because they cancel each other. The arithmetic mean would treat it differently because in absolute terms 10 seconds slow down is bigger than 5 seconds speedup. But that is not fair for speedups because the absolute speedup you can reach is at most 10 seconds but slow down has no limits.

This is the best explain-like-im-5 I've heard for geo mean and helped it click in my head, thank you :)

But reality doesn’t care:

If half your requests are 2x as long and half are 2x as fast, you don’t take the same wall time to run — you take longer.

Let’s say you have 20 requests, 10 of type A and 10 of type B. They originally both take 10 seconds, for 200 seconds total. You halve A and double B. Now it takes 50 + 200 = 250 seconds, or 12.5 on average.

This is a case where geometric mean deceives you - because the two really are asymmetric and “twice as fast” is worth less than “twice as slow”.

There is definitely no single magical number that can perfectly represent an entire set of numbers. There will always be some cases they are not representative enough. In the request example you are mostly interested in the total processing times so it does make sense you use a metric based on addition. But you could also frame a similar scenario where halving the processing time lets you handle twice as many items in the same duration. In that case a ratio-based or multiplicative view might be more appropriate.

Sure — but the arithmetic mean also captures that case: if you only halve the time, it also will report that change accurately.

What we’re handling is the case where you have split outcomes — and there the arithmetic and geometric mean disagree, so we can ask which better reflects reality.

I’m not saying the geometric mean is always wrong — but it is in this case.

A case where it makes sense is what happens when your stock halves in value then doubles in value?

In general, geometric mean is appropriate where effects are compounding (eg, two price changes to the same stock) but not when we’re combining (requests are handled differently). Two benchmarks is more combining (do task A then task B), rather than compounding.

Squaring is a really good way to make the common-but-small numbers have bigger representation than the outlying-but-large numbers.

I just did a quick google and first real result was this blog post with a good explanation with some good illustrations https://jlmc.medium.com/understanding-three-simple-statistic...

Its the very first illustration at the top of that blog post that 'clicks' for me. Hope it helps!

The inverse is also good: mean-square-error is the good way for comparing how similar two datasets (e.g. two images) are.

The geometric mean of n numbers is the n-th root of the product of all numbers. The mean square error is the sum of the squares of all numbers, divided by n. (I.e. the arithmetic mean of the squares.) They're not the same.

I'm not gonna edit what I wrote but you are interpreting it too way too literally. I was not describing the implementation of anything, I was just giving a link that explains why thinking about things in terms of area (geometry) is popular in stats. Its a bit like the epiphany that histograms don't need to be bars of equal width.

Reply on news.ycombinator.com