Groundhog Day - or the Problem with A/B Testing | Hacker News Reader

Groundhog Day - or the Problem with A/B Testing

codinghorror.com

65 pointsfogus17 comments

I wonder if the conclusion about A/B testing would have been the same if the tenuously-related lead-in anecdote had been about medicine ("Although randomized drug trials might have succeeded in saving billions of lives, it didn't ring true to patients...")

You can ask users about their satisfaction. Do it. If you do A/B testing well, after testing it goes up. I'm sorry if it doesn't provide spiritual fulfillment to Jeff Atwood or product visionaries who hate the idea that something so simple so often trumps their vision, but surely you can work through that after you see the benefits available. A/B testing let me help over 100,000 extra kids learn to read this year -- that is pretty freaking spiritually compelling.

Hi Patrick...

I'd just wonder if it would be safe to say that A/B testing is a good tool..., but should be used in conjunction with other tools such as knowing-your-customer, intuition, expertise etc.

I'm not in a space where I can A/B test anything but from the outside it seems as if A/B testing can be mis-used as a replacement for traditional methods and can be obsessed about because it IS measurable.

The local minimum point is correct I think. Finding the true maximum of a highly multidimensional quantity is fiendishly difficult. If A/B testing were sufficient then so many human and universal problems would vanish.

So I totally agree with the article: you can polish with A/B, but unless you're pretty happy with the neighborhood you're in it's not the only tool in your belt.

A/B testing is great, but your data is severely limited. How do you know that sticking with the vision doesn't pay off in the long run? Unless you've been running A/B tests for years with doppleganger-Patrick who eschews the cold, mechanical methods of A/B testing, writes blog posts about how important gut-feeling is in web design, and made every page of BCC have yellow text on a green background because dammit he likes John Deere, how could you know that you're better off now? We know a lot less than we think we do, and running some limited tests doesn't change the fact that most of the time we're just guessing.

The problem I have with valuing A/B testing over the visionary is that A/B testing is a tool, not a purpose. The lean startup movement is everywhere and I keep seeing newbies coming onto the scene, reading a few articles and going "I don't really have an idea or a passion, but I'm going to A/B test my way to $1M business all by myself. But wait... how do I start?"

It takes vision to decide what business to start and passion to follow it through and creativity to figure out what to try next. A/B tests -are- useful and they do have a prominent place in a business but please let's not pretend that's all it takes to build and grow a successful business. Look around at the founders of both VC backed and bootstrapped successful businesses and the common threads are much more about passion and persistence than the ability to A/B test.

A/B Testing is a manual hill-climbing algorithm, and we can't see the hill. Imagine a vast 2d plane that represents all the possible combinations of your website - its buttons, colors, copy, layout. There are mountains and valleys that represent spots of high-signup and low-signup. You are at a certain point, and an A/B test will, say, move you North, at which point you can see what your altitude is. Is it higher? Lower?

Unfortunately moving like this will trap you on a particular mountain, until someone comes along with a helicopter, saying - hey! I think I see a bigger mountain range over there! - and you try out a completely different design. Which you could certainly A/B test to see if the spot you landed on is actually higher than where you were (which if you're at the base of a huge mountain, may not be the case!).

Of course, the combinations are multi-dimensional, and so are the fitness functions (signup, retention, word-of-mouth, etc). But I've found this useful to explain why A/B testing has problems; that is to say, it has specific uses, and so do UX designers.

I think Atwood misses the point a bit here: A/B testing is not destined to find shallow local maxima; one can also test larger, more significant differences.

And, ultimately, Phil (in Groundhog Day) discovers that.

Right, there had been similar arguments by Rand of SEOMoz and Jason of Smartbear.

A/B testing doesn't dictate what you are testing. It is an instrument -- ultimately you have to plan the experiment. A lot of people confuse that testing is just limited to doing red v/s green button color test. But what prevents you to converting your site to flash app and then A/B testing it. What does it have to do with creativity?

Though all such articles are good at at least generating awareness about A/B testing.

The problem, of course, is that you can only afford a certain number of big experiments. Life is finite.

And big experiments can be hard to A/B test against your mature product. This is what that whole "chasm" thing is about: the audience for new things is different from the audience for old things; present a radical new thing to someone who isn't an early adopter and it will get low marks. The automobile had really bad A/B results against the horse, for most audiences in the late 1800s. It is hard work to attract a new audience, when you have the alternative of incrementally improving the experience of the old audience.

And, of course, the more time you've invested in building your radically different idea, the more crushed you will be by bad A/B data. A/B testing of tiny differences that can be toggled in five minutes is the least emotionally painful form of A/B, so no wonder it is so much more popular, to the extent that Atwood thinks it represents the entire field.

Atwood's metaphor is still perfect, however. The movie Groundhog Day covers all of this. It's an astonishing work of art.

I've heard that life is like a box of chocolates. I've also heard that stringing together aphorisms, shaky analogies, and vacuous statements doesn't make a coherent argument, but the size of Jeff Atwood's audience suggests I'm wrong. More A/B testing for me?

Look, A/B testing is a tool using statistical tests of significance. These statistical tests are based on a bunch of assumptions, the most important of which is (usually) independent and identically distributed data (i.i.d.) If your data isn't i.i.d. (like, say, the changes in opinions and attitudes that occurred over the many years during which the automobile was developed) the conclusions reached by the test don't hold. Still, not investing in automobiles was probably a good investment unless you got lucky.

I disagree with that quote he throws out at the end there about A/B testing not being able to construct things. Isn't that essentially what genetic algorithms do? Granted, we don't use that sort of process to make a website because it would be slightly insane, but who knows?

And even starting from one sub-optimal site, isn't arriving at some better, more refined experience a form of construction? If you're not discovering something unexpected or seeing something new form from A/B testing, I think that speaks more to your lack of imagination or unwillingness to try than it does some inherent limitation of testing.

Genetic algorithms do create things. It's the randomness seed at the start that creates the new thing.

Interesting article. I was fascinated by his point on Groundhog Day, then I disagreed with his point on A/B Testing (because I used to feel the same way), and then finally I loved this quote at the end - "A/B testing is like sandpaper. You can use it to smooth out details, but you can't actually create anything with it."

I can't think of another article that short on a mundane topic that I had three distinctly different, relatively strong emotions to.

For anyone that doesn't know it (like me), he mentioned the Ouroboros. I found it pretty interesting. From Wikipedia:

"The Ouroboros or Uroborus[1] is an ancient symbol depicting a serpent or dragon swallowing its own tail and forming a circle."

http://en.wikipedia.org/wiki/Ouroboros

It's (probably) a Red Dwarf reference.

I wonder why would anyone want to give up reliving the same day after measly 30-40 years. Would it be better to age in the meanwhile?

It's a movie. In the real world after having the perfect date the woman ends up loving you forever.

Reply on news.ycombinator.com