Hacker News Reader

Can you say more about how your example copy change experiment had a confirmation bias? I’m guessing the issue was it was inconsequential and you happened to see an uptick in the initial hours but it would have evened out over time. But hearing more about that would complete the case study.

View on HN

Yeah exactly, it was some sort of seasonality effect that, or just plain randomness.

But you showed an example which was significant. That confuses your point somewhat.

It’s also not clear why you think “most” startups are not doing statistical significance tests on their a/b test results

Reply on news.ycombinator.com