Benchmarking software is so easy to do wrong. It's so easy to manipulate the results if you are so inclined, and even when you're trying to be evenhanded it's so easy to disadvantage one of the systems under test.
Different approaches are called for with different tools, and the classic error we see over and over again in benchmarking studies is the inappropriate application of idioms which suit one of the competitors across the others.