Implications of use of multiple controls in an A/B test
blog.twitter.com
blog.twitter.com
I wonder what intuitions drove that suggestion? Would they be even happier with 3 buckets (each a third of the size)? With 4 (quarters)? 5 (fifths)? ..._n_ buckets, 1 for each sample? At what point did those experimenters think it would stop being an improvement over a single control group?
Re: N buckets: you'll get diminishing returns and it will take N times more data points to get a statistically significant result in any bucket.
[1]: http://www.evanmiller.org/how-not-to-run-an-ab-test.html
This is a net loss.
One of the big issues with experiments is how much value they really drive, which is the real effects. In many situations we do A/B testing for changes whose real impact is a fraction of a percent of the tracked metric. If anyone followed the discussion about p-hacking in science [1] should really start wondering if the way we run experiments, considering the little changes we see, is not nearing p-hacking, especially when I see this ideas floating, and if we don't need to find more significant way of running our experiments.
[1] One of the first article I found about the issue http://journals.plos.org/plosbiology/article?id=10.1371/jour...
We covered some of this, at a high level, here: https://blog.twitter.com/2015/the-what-and-why-of-product-ex...
Certainly not everything needs to be A/B tested -- depending on the nature of the change and the kind of insight one is looking to get, MAB may be better, if it applies, or a number of other approaches can be more beneficial, including not running an experiment at all. After all, Twitter itself came from a side project at the podcasting company Odeo. If Twitter was a/b tested, I'm sure it wouldn't move any podcasting metrics in a meaningful way :).
I'm obviously biased -- but I've seen really important insights gained from A/B tests, and wouldn't dismiss them too easily.
I know others will disagree with me, but my feeling after working in a number of jobs in which large systems were built to automate large-scale frequentist hypothesis testing operations is that these methods just wholly fail, and that business politics leads people to read whatever conclusions they'd like to from the results, and give credit back to the statistical framework even if it is not deserved.
I should be careful not to over-state this too much. I understand there is a mathematical framework for these tools and despite being trained in machine learning and Bayesian methods, almost all of my professional experience has been working with frequentist statistics. Sometimes you have enough data that it would probably swamp out whatever prior you might choose; often you don't have (or want to invest in getting) expert priors, and "non-informative priors" and that ilk sort of rope in some of the same problems as what frequentism has. You can get a limited amount of evidence from doing simple frequentist things, but you also open the door for all of the misunderstandings of p-values, ignoring effect sizes, doing model selection by directly comparing t-stats, etc., which just seems to be way, way less of a problem if you use other methods.
Some of the main sources of enlightenment for me on this have been:
[0] http://lesswrong.com/lw/g13/against_nhst/
[1] http://www.columbia.edu/~gjw10/achen04.pdf
[2] http://www.indiana.edu/~kruschke/articles/Kruschke2012JEPG.p...
The last article, by Kruschke, spends a lot of time on exactly the type of issue from the Twitter article -- something like two-group mean testing.
In general, I cannot understand why people continue to use frequentist methods on policy problems -- that is, problems where you are inherently looking for Pr(Hypothesis | Data), and where, due to all of the many, many pitfalls from the above sources and elsewhere, you can be sure that Pr(Data | Hypothesis) is not at all a suitable proxy for Pr(Hypothesis | Data).
I would view any labor costs associated with figuring out why A/A/B testing is not a useful idea to be pure waste, not because the math is wrong or anything (the math is interesting and good!) but because, from first principles, this just seems like totally and completely the wrong way to approach this kind of applied problem.
And yet, everyone is doing it, and baking it right into their huge production analytics systems -- most notoriously all of the modern fly-by-night marketing tech startups.
I don't necessarily disagree with you! Though I hope you are not calling Twitter a fly-by-night marketing startup :).
As a company, we've found A/B testing very valuable, but it certainly has limitations, and pitfalls that one must be fairly experienced to avoid. You can see some descriptions of what we do to achieve that in other posts in that series. But it's fair to say that this takes some work (like the work described in the "second control" post) to avoid badness, and it's fair to question if other methods can get us to the same end result -- ability to evaluate impact of proposed product changes -- cheaper, or more accurately.
That remains to be seen.
At the end of the day, we need to be able to tell management that we believe the change had this big of an effect, within these errors bars. Management need that in order to make financial decisions. They aren't dumb, but they also can't be expected to have a masters in statistics.
In general though, I would dispute a lot of your claims about what industry statisticians (or "data scientists" if you want to use a buzzword) need to do. One of the primary jobs of industry statisticians, it would seem, should be to slow everyone down on purpose, because everyone is in such a rush to get "insight" and "actionable information" and to get some sweet data viz of some KPIs that they are doing horribly shoddy statistics, and mutating the craft of statistics overall into some kind of shoddy data journalism / data story telling circus where the careful avoidance of statistical pitfalls doesn't seem to matter to anyone.
What management need is the answer to what to do, as quickly as possible. If they pick the wrong answer because of bad statistics, the company loses to the competitor who randomly picked the right answer. If the company makes no decision quickly enough, the company loses to the competitor who quickly randomly chose the right answer.
So given those constraints (the need for accuracy and the need for answers as soon as possible) what is the best way forward?
Remember: they don't pay us because math is fun, they pay us to solve their problems.
Also, none of this is nearly as binary as your comment makes it seem to be. Yes, you might lose a little in the short-term to a faster, but more error-prone, competitor. But as long as the loss isn't so severe that you become insolvent, then you might live to see the day when all of the competitor's errors and sloppy work come home to roost and they fail spectacularly. In general, doing it wrong but fast is pretty worthless, investors and customers will catch on to you and bail, and only the highest level of management will have insulated themselves from the losses that result.
Working for a steadier company that prioritizes building a name based upon actual quality, and is willing to tell customers or investors no when they must do so, to regulate feature creep, ensure high quality standards and accuracy, and so forth, is not only much more rewarding, but also more likely to lead to stable income.
I think the type of binary, winner-must-take-all-as-soon-as-possible thinking is very destructive. It's not a useful mental model of how this stuff actually works, but because so many managers believe that it is, they engage in all kinds of unnecessary and self-destructive anti-quality patterns, and they end up damaging everyone in a tragedy of the commons sort of way.
To some extent, this is inherent to the application of the tools. In an A/B test, done by a business, it is incomplete to apply 'just' a frequentist hypothesis test (whether or not you fall into any of the traps) or just a Bayesian test with a relatively arbitrary threshold like 99% probability of improvement. As has been pointed out many times, your standard A/B test is focused solely on precisely whether there is an effect or not, while ignoring all the costs and benefits; which means you get precise expensive estimates of the wrong thing. You need to broaden the picture and explicitly acknowledge that it's a decision theory problem and this is why multi-armed bandits are the right framework for A/B testing.
Unfortunately, it takes so much time and effort to understand all the statistical material about testing and power and binomials and whatnot that anything to do with decision theory tends to be relegated to advanced courses or later textbook chapters, if people ever hear of decision theory (or reinforcement learning, or sequential trials, or multi-armed bandits) at all. You might consider decision theory to be the flower of statistics, in which all the cookbooks and algorithms reach their justifications and you know when you can break the various rules of thumb you've learned, but it requires so much time to teach the biology of plants that few students get there or are still interested.
I'm trying to consolidate decision theory materials at https://www.reddit.com/r/DecisionTheory/ (submitted at https://news.ycombinator.com/item?id=10897324 ) if you want to read up more on this topic.