A/B Test results: Trying different homepage heroes got us 2.6x sign-ups

Last year, we rebranded forms.app, a form builder app. With the rebranding, we had an experimental hero section that we quite liked. It was an AI generator; people write what type of form they like, and even before they sign in, they get it.

I'd have liked to include all images here, but you can check them out here: https://salimdin.substack.com/p/how-an-ab-test-more-than-doubled-our-signups-on-formsapp

After a few months, we realized something was off. We thought it was about rankings, but after a few discussions, we decided to test our homepage hero section against two alternatives.

- Variant A: The AI generator version

- Variant B: Only sign-up buttons (nothing else)

- Variant C: A single button for the app itself (playground without sign-up)

We ran the experiment for 2 weeks and only for new visitors. The results were interesting, to say the least:

- Variant A came last on every single metric

- B got 2.6x the sign-ups of A

- C got fewer sign-ups than B, but those who did sign up activated more; so it's a more distilled result.

Seeing these results, we decided to move forward with Version B, as it has more promise, and agreed that B and C deserve their own testing.

Recently, we changed our pricing, and with it, Version B got an update. Soon, the C version will be updated too, and we'll pit them against each other. I'll be sharing the results when we run that A/B test.

What do you think? Are the results what you expected, or did they surprise you too?

It's not a promotion, just an article where I show the version and the results, but I couldn't submit with the URL (probably due to the Substack URL). So here it is: https://salimdin.substack.com/p/how-an-ab-test-more-than-doubled-our-signups-on-formsapp