Hacker News Reader

Benchmarking anything on EC2 instances will yield statistically suspect results. You know nothing about the other workloads on the box, the underlying hardware, or the network connectivity.

View on HN

Doesn't mean it can't be statistically valid, the noise just means you need to run more tests.

That assumes the noise is uncorrelated. It's not hard to imagine scenarios where it's correlated.

Reply on news.ycombinator.com