My performance benchmarking workflow
blog.vmsplice.net
blog.vmsplice.net
If you’re running VMs in production, VM performance is the main metric you care about. If not, benchmarking in a container limits performance interference to networking and disk, mostly.
Performance measurements are supposed to be like “real science” experiments. Doing it the “right way” would have controls, multiple trials, and different setups to help control for measurement effects. Unfortunately, most experiments do not have sufficient rigor. The post shows how to document experiment setup, which allows you to replicate, but it may not actually be able to reproduce the hypothesis/conclusion.
Just to add on: the software environment should be controlled obviously, but the actual environmental conditions around the machine also need to be controlled for an accurate measurement.
https://benchmarks.ul.com/support/how-to-benchmark "For example, you should test every system in the same location, at room temperature, and away from direct sunlight and other heat sources."
And even after all that one should expect a 3% spread in scores.