http://www.evanmiller.org/how-not-to-run-an-ab-test.html gives a good explanation about how this can ruin your tests.
FTA: Bayesian experiment design: With Bayesian experiment design you can stop your experiment at any time and make perfectly valid inferences.