Hacker News Reader

Comment by noxvilleza | Hacker News Reader

Is there an existing named adage for something like "if one creates a benchmark in order to rank general performance of some products, some of those products will ultimately sacrifice general performance in order to optimize for that benchmark"?

View on HN

https://en.wikipedia.org/wiki/Goodhart's_law

You stated it almost directly.

Oh dear. I did a brief search for 'adage on benchmarking' and only saw Rugg/Feldman benchmarks.

It's also why the only true benchmark is using the thing as it needs to be used - but this is hard to compare because often you need code to work with the tool and vice-versa.

there are the TPC benchmarks which try to cover a wide variety of use cases and scenarios and are designed independently from any one engine: https://www.tpc.org/information/benchmarks5.asp

you post the results for your own product, others do the same, customers can compare: https://www.singlestore.com/blog/tpc-benchmarking-results/

It is partially true, but this benchmarks force schema. You can't reorganise data for example in wide table or add indices. So it actually does not show you how to use the system to solve this type of problems in a best way possible, but checks unoptimised results as if you never learn and never utilise best practices of the DBMS you choose for production.

as long as vendors are presenting the results of their own products, I'd expect them to de-normalize and tune the queries to leverage every advantage their engine can possibly bring to bear on the benchmark's problem set - so long as they're transparent about what they did. Clickhouse apparently does quite well on TPC-DS according to: https://aavin.dev/tpcds-benchmark-on-clickhouse-part2/ but I'd love to see a more official result that the CH engineers would stand behind.

Reply on news.ycombinator.com