The temptation to "peek" and keep on peeking until the test confesses to the thing you want it to say is very high.
The temptation to "peek" and keep on peeking until the test confesses to the thing you want it to say is very high.
If you peek, you really have to commit to running the test for the full duration no matter what.
It's more complicated, but you can also run sequential A/B testings using [SPRT](https://en.wikipedia.org/wiki/Sequential_probability_ratio_t...) or similar, where a test gets accepted or rejected once it hits a threshold. I won't go into the details, but you can incrementally calculate the test statistic, so if your test is performing very badly or well, the test will end early.
One product team I worked in run all tests as sequential tests. If you build a framework around this, I'd argue it's easier for statistics-unaware stakeholders to understand when you _can_ end a test early.