Hacker News Reader

Nah because we care about the size of the effect too.

We can get a p<0.0001 finding of an otherwise terrible, annoying change that saves 1%.

That’s a more subjective task of balancing trade offs: how much it saves versus how annoying the change really is. It’s not the job of the benchmark utility to make these subjective decisions.

But I really care about "there's a 95% chance the benefit is somewhere between 1.9x and 5x, so it's worth it."

So that's why CIs are interesting and p-values are less so.

Reply on news.ycombinator.com