610 karma · joined May 4, 2011
Evan Miller has a great sample size calculator to illustrate this: http://www.evanmiller.org/ab-testing/sample-size.html
In the example above, Google's MDE is 0.005% and the small startup's MDE is 60%. If the small startup's baseline conversion rate is 5%, then with 95% confidence and 80% statistical power, they can reliably detect a 60% effect (positive or negative) with 1,968 visits.
It's not just the 60% increase that you should be mindful of. I bet you all small startups want to know if the change they made to their homepage decreases the conversion rate by 60% or more. With an a/b test, they can do that.
Another point to keep in mind is that you can learn a lot from a statistical tie (not enough data to conclude there is a difference). In fact, no matter what your traffic levels are, most of your experiments will be a statistical tie. It's important to learn what doesn't work just as it's important to know what does work. This can really help you with prioritization. In the example above, the small startup can use the statistical tie to draw the conclusion that some tasks in their product roadmap will result in less than a 60% effect in the conversion rate and can be prioritized as such.
But ultimately it will be quite difficult for the small startup to produce variations that have at least a 60% effect in the conversion rate, but it's not unheard of and again, you can learn from a statistical tie.
Just because you have low levels of traffic doesn't mean you can't learn anything from a/b testing.
Optimizely is the best a/b testing software I've ever used. It's flexible and easy to use. I suggest you check it out.
The only difference between the two pages was the page load time. Because we removed all other variables (time, appearance, functionality, etc.) we can say that the 60% performance increase lifted conversions by 14%.
As for employer and occupation, we are legally required by the Federal Election Commission to do our best effort to gather the donor's employer and occupation. Doesn't have anything to do with taxes.
Yes, the donation submission was an POST request executed with JavaScript (AJAX) from the user's browser.
Thanks for noticing the bug! I'll have to check that out.
With most CDNs you have what's called a pull zone that points to your origin server on a different domain. So when the user requests something like cdndomain.com/whateverpage the CDN looks for the file at pullzonedomain.com/whateverpage and then it serves what it gets back from the pull zone.
Akamai has more advanced configuration options than Amazon Cloudfront. One example of this is that our payment processors did not actually live on contribute.barackobama.com, only the static files did. So we used Akamai to setup a reverse proxy to POST the donations to a different domain. Without this reverse proxy it wouldn't have been possible because the same origin policy forbids cross domain, client-side POST requests.
Obama for America (OFA) is hiring many frontend developers to work on barackobama.com.
We are essentially a large startup that hit the ground running in April of this year. We need skilled frontend developers to help us create another groundbreaking information and mobilization site to help re-elect the President.
We're looking for developers with experience in HTML5, CSS3, JavaScript, jQuery, jQuery mobile, ExpressionEngine, social APIs, Git, A/B testing and anything else that is necessary to achieve our goal.
OFA is a very unique opportunity to work for the President on a high profile site that has tons of traffic. One of the many benefits is working with a very talented design team.
If you would like to have a part in helping the President strengthen the country and improve the lives of millions of Americans, contact Kyle Rush (@kylerush OR krush@barackobama.com). Job description/posting here: http://www.authenticjobs.com/jobs/9256/frontend-developer