HNHacker News
TopNewBestAskShowJobs

aisscott

17 karma · joined July 22, 2018

submissionscomments
aisscott··on p-Hacking and False Discovery in A/B Testing
Hi, I am one of the authors. We found that people p-hack with traditional t-tests. Most A/B tests were run this way in the past and some still are. The paper is using Optimizely data (from 2014) before Optimizely introduced new testing in 2015 designed to solve the issues we found in the paper.

If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://www.optimizely.com/resources/stats-engine-whitepaper...