1,724 karma · joined March 21, 2011
Several of our A/B tests have used total size but it doesn't have a big enough effect on conversion to be measurable.
As for sample size, we're vigilant to make sure that we reach statistical significance. Typically our tests will encompass about 2,000 data points per group.
Our job is made easier by the fact that we're looking for huge wins so when we see a 10% gain with a 99% confidence, that's still a "fail" scenario for us.
We expect that we'll rapidly exhaust the supply of radical ideas and then we'll be left with two or three winning UX paradigms. From there we'll begin doing "normal" A/B testing in a linear fashion.
I'm going to restart the tests using three domains and let them run over an extended period so that I can see whether I am, in fact, getting users who're searching for meals vs rentals.
The numbers were crystal clear though: bounce rate was the same for all the domains indicating that users were getting what they expected after clicking through.
In this alternate reality, I began the experiment on January 2 and ended it on January 8. Would that render the invalid experiment valid?
Our criteria for picking the winner came down to one number: tour request rate. In other words, what percentage of users found a listing they liked and contacted it using a particular design. We use other data points to augment this core metric (bounce rate, time on site, number of listings viewed, search criteria revisions), but ultimately a variation only wins if it's better than the control at getting users to an office space that they like.
We do get spammed but we've been able to keep a handle on it. Mailgun's filters capture most of the spam and we have backup handlers in place to address the rest.
For example, the ternary operator is consider by many to be an elegant solution to simple if statements but from a comprehension (and therefore bug-finding) standpoint, is it superior? Also, how does this effect change as the size of the codebase grows from a single page (as depicted in the video) to a more complex class file.
value = test ? (some_value * multiple) : false_value
# vs
if (test) {
value = some_value * multiple
} else {
value = false_value
}Those guys help a lot when you've only got 24 hours to get over the hump (ask the Firebase guys).
If, for example, the US were borrowing money at 3% by issuing T-Bills, it would just buy a basket of high-grade foreign and corporate debt at 3.5% and earn a spread of 0.5% while continuing to provide adequate liquidity of T-Bills.