If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://www.optimizely.com/resources/stats-engine-whitepaper...
If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://www.optimizely.com/resources/stats-engine-whitepaper...
All that you can really do is prove it wrong, by measuring an effect when there "should", by the hypothesis, be none.
Due to what is known as the "problem of induction", it's not sufficient to accept a hypothesis because you appeared to not measure an effect in the past, as that says nothing about whether an effect will occur the next time a measurement is made.
p-value is the "chance" of measuring an effect, given that no effect actually occurred.
This can also be done nicely with a Bayesian mixture model or a spike-and-slab multilevel model, and that is what is done in "What works in e-commerce - a meta-analysis of 6700 online experiments", Brown & Jones 2017 http://www.qubit.com/sites/default/files/pdf/qubit_meta_anal... (although they don't formulate it in terms of a sharp null but ask the more relevant 'probability of a >0 beneficial effect', which for some kinds of A/B test has a very low prior - like only 15% for 'back to top' A/B tests).