Hacker News Reader

Comment by vtail | Hacker News Reader

The author seem to be implying conversion loss from pre/post which may be affected from many factors including seasonality - I’m wondering why didn’t he do an A/B for a radical design like that?

View on HN

I did actually split test. I had traffic going to both types of our landing pages. I didn't like the AMP results but there wasn't enough traffic yet to draw the conclusion, so I just put all the traffic there and measure across time periods a bit.

As for seasonality, that's a fine argument. But overall traffic was consistent across these weeks measured. Also, our conversion rate was awesome through US holidays like July 4. Also, as soon as I got rid of AMP, immediately our conversion rate is back. It's pretty stark how bad AMP was for us.

Ah ok that make sense. Not sure how you measure conversion, but in our case (I’m in B2C ecommerce) both traffic and conversion often fluctuates, so if one is stable doesn’t mean that the other won’t move.

Can you A/B test a site like that? I don't really know how AMP works (so I may be way off base), but from a user standpoint, since you don't seem to actually go to the page, just to Google's AMP version, I'd imagine there's no way to control whether the user is getting an AMP version or not in an A/B way. It seems like you're either AMP enabled (and the user gets AMP) or you're not (and they get the normal page).

Yeah, it's not easy to test AMP unless you're doing things like bringing in traffic from Adwords where you can split the traffic up. I do a bunch of testing with that Adwords traffic.

Reply on news.ycombinator.com