Hacker News Reader

Instead of giving users a confidence interval, ask the user to specify a p value. Then run something like Welch’s t test.

View on HN

Nah because we care about the size of the effect too.

We can get a p<0.0001 finding of an otherwise terrible, annoying change that saves 1%.

That’s a more subjective task of balancing trade offs: how much it saves versus how annoying the change really is. It’s not the job of the benchmark utility to make these subjective decisions.

But I really care about "there's a 95% chance the benefit is somewhere between 1.9x and 5x, so it's worth it."

So that's why CIs are interesting and p-values are less so.

Reply on news.ycombinator.com