Ask HN: A/B testing with only 1-2 sign-ups per month

2 pointstimmorgan4 comments

I would like to employ A/B testing for my web service homepage and sign-up. My site only gets 1-2 sign-ups per month.

How long would I need to run the experiment to get a reasonably decent idea of which version is statistically better? (I have no statistical background/training.)

The wikipedia page (http://en.wikipedia.org/wiki/A/B_Testing) is very light on details.

Any help would be greatly appreciated.

View on HN

Here's a great post on doing the math for this stuff

http://blog.asmartbear.com/easy-statistics-for-adwords-ab-te...

But yes, the bad news is it's going to take too long at your rate right now. I recommend looking at things like http://conceptfeedback.com to get feedback in lieu of being able to test right now. You could also consider running a poll with Mechanical Turk workers. The turkers would answer a poll like which design is better for about a $0.05 a person.

I've done the conceptfeedback.com for testing some new elements in http://tgethr.com before we had the traffic to start experimenting and the feedback was great.

Wonderful advice. Thank you.

Based on your current traffic? About 55 years.

So you're telling me there's a chance... :-)

Reply on news.ycombinator.com