Hacker News Reader

Comment by samd | Hacker News Reader

I don't know what's more surprising, that one small change is worth millions or that A/B testing is still news to some companies.

View on HN

I have spoken to a lot of companies who have A/B testing somewhere on their radar, but some combination of "That's nobody's job, so we haven't actually started doing it", (sometimes) organizational infighting (marketing wants to start, engineering has higher priorities and refuses to allocate resources to "twiddling text", etc), and uncertainty about outcomes means they don't start it yet.

One of my big tasks at some consulting clients is convincing people to buy into A/B testing. The easy way is to show them how their website is leaking staggering sums of money, and present A/B testing as the cheap, reliable solution to that problem in the future.

These are smart software companies who transact most of their business online, by the way. The broader economy which isn't primarily web-focused has many, many, many opportunities to improve their execution.

Wasn't there a famous post a few years back by amazon about their a/b tests resulting in 60million in sales increases, just by changing the button colors?

Everyone working on ecommerce sites of this scale should be expected to know about this stuff

This wasn't A/B testing. This was fixing UI mistakes based on UX analysis. There was no 'B'. They weren't comparing A and B.

UX analysis and A/B testing go hand in hand. There'd be no way to know that the Company field was a UI mistake if it had not been tested.

In this case, the A/B testing would have been with that field and without.

How would you know which fields to use?

Yup, it's shocking. However, is this more of a problem with A/B testing or just bad user experience/design? I suppose an A/B test would've been to remove the Company field from the payment process and see how things have performed.

I think companies are more concerned with more obvious A/B tests (button color, page headings) than with testing large changes to their site or testing more in depth things like user experience.

A/B testing probably wouldn't have caught this.

I don't know what's more surprising, that one small change is worth millions or that A/B testing is still news to some companies.

Yeah, my company has been doing A/B (we call it Champion/Challenger) testing for years.

A/B anything (testing or dual projects at once etc) is seen in some places as a "lack of skills, otherwise you would know beforehand".

(which I don't agree with, in case someone doubts!)

Reply on news.ycombinator.com