Hacker News Reader

Comment by omgtehlion | Hacker News Reader

> Intel i7-12700K processor

Wait! You can’t benchmark on a CPU with unpredictable speed and mix of slow and fast cores. All kinds of effects come into play here, and the code itself is not the most prominent among them.

To measure which _code_ is better, you should use real SMP machine with fixed clock speed and turned off HT. On the machine from TFA you are just fighting with Intel’s thermal smarts and OS scheduling shenanigans. (edit: you can use the same machine, but configure it in BIOS. I, myself, use i9-12900k fixed to 5ghz@8P-cores as "Intel testing machine")

View on HN

Except then you're not testing on the hardware configurations people will actually run the software on.

People do run software with HyperThreading on and on E cores and with Turbo boost and throttling. Having code that behaves better in these dynamic, heterogeneous environments is a net benefit to the user.

youre missing the point. you need to remove as many variables as possible when benchmarking because benchmarks are always relative to each other.

in absolute terms testing is required on a standard config, but benchmarks should mostly be conducted with a minimum of external variants if you want to draw meaningful conclusions

It is easier to deal with all the issues separately. First your algo (in single thread), then threading, then memory accesses and cache effects. After you sort everything in your control, you try to deal with the real world (like counting only “real” cores, ignoring/or enjoying hyperthreads, OS scheduling, but most of these are quite unpredictable, and if you get your code run faster in 12700k, the same setting will be slower on, say, 5800X or on server machines, while basic stuff speeds up the program on any hardware).

Those issues are not separable. The performance of memory and caches can determine which algorithm is best. This is the basis of the entire field of cache-aware algorithms, which routinely beat the pants off of theoretically superior algorithms.

In my experience (which is in video transcoding and in research astrophysics - both domains where it matters!), if you really need to squeeze out performance, you have to design with the target platform available for profiling and benchmarking from the beginning.

Edit to add: I agree wholeheartedly with your top-level comment! I just am perhaps more extreme than you; I don’t think “laptop benchmarks” can ever be twisted into being useful.

You usually don't design an algorithm for a single CPU. Most software has to run on different tiers and generations from Intel, AMD, a whole herd of different ARM processors. Even the very hottest code paths will mostly do CPU feature detection but not "if cpu == i7-12700K { ... }"

I get what you are saying, but in some domains, you really do design for a single SKU, beyond even just the CPU. Supercomputers and computing centers like SLAC have a very constrained set of SKUs that your software will ever run on.

I know this is not how most cloud or consumer software works, but that stuff usually isnt as performance-sensitive.

There's plenty of CPU-cycle-munching work in the pro segment that does care about performance and can't assume fixed hardware. CI, compression, CGI, ...

«Most software» - lots of software is run on cloud compute / backend servers and can be built or tuned specifically for the box/vm/container you’re running it on. I would expect AWS/Azure/Google to do that for their PaaS offerings, for cost savings.

I barely even try to run micro benchmarks on my work laptop, and I’m the guy who introduced them. It’s just so random.

Pretty much I’m looking for a slowdown that’s more than 25% and anything less could just be noise. When you’re reaching for 5% improvements, that means you have to let the CI system tell you, and production response times are the final arbiter of truth.

Do you have a system for running benchmarks in your CI?

I did a reply and then I somehow lost the whole thing. Nodejs has microbenchmarking tools. They are borrowed from other languages though.

Mostly what I did was cook the results to send them to telemetry. Not unlike exporting your unit test results.

Software should be tested on a system like the one that will run it. All modern machines use dynamic frequency scaling. It should rarely be disabled in production. Many modern machines have heterogeneous cores. Your testing procedure might be reproducible but it's unrealistic.

Reply on news.ycombinator.com