Hacker News Reader

The cult of statistical significance is alive and well. A 0.05 p-value implies a 1:20 chance of "alternative" performing worse upon final installation. That's rather risk adverse. It also implies that "alternative" is worse from the get-go. When is that the case? Type 1 and Type 2 errors are much more balanced in web apps. Anyone care to show me why that's a bad mentality?

View on HN

No, it doesn't. It means that there is a 1 in 20 chance that you would have seen results as good or better if your change had no effect (assuming a standard one-sided hypothesis test). Thus if the effect appears to be good, you should take the test as some evidence that it is worth implementing.

Right, that's a specific use, but I'm speaking of a two-sided test where you're indifferent between alternatives.

Reply on news.ycombinator.com