Hacker News Reader

Comment by Nadya | Hacker News Reader

I'm one of the people who blacklist websites that do this. If you give me an annoying popup halfway into reading your article I close the tab and blacklist your site from my reading materials. So when presented with an opportunity to people in charge of websites doing this - I asked if they had actually A/B tested these to see if they had any merit or if they hurt user retention.

Sadly, they work! One claimed a 78% larger sign-up rate and the other was something like a 65% larger sign-up rate.

I did not discuss churn rates with them - but overall traffic to the sites weren't largely impacted I was told.

Note that the popups on each of these sites was a one-time deal on a user's first visit. After closing it you never saw it again (unless you clear your cookies or browse incognito, etc.)

View on HN

> One claimed a 78% larger sign-up rate and the other was something like a 65% larger sign-up rate.

Important, but unanswered questions (because they are hard, so let's ignore them!):

(1) How many of these users just mark all of the newsletters as spam?

(2) How many users don't realize that they can dismiss the dialog, and enter their email because they think that they are forced to? How many of these are really "engaged" users vs. just users that are willing to jump through a couple of hoops before they disappear in the ether?

(3) If those extra sign-ups are due to people thinking that they have to enter their email, how is this any different (ethically or practically) than actually forcing them to enter their email? Are you only making this distinction in the smallest way possible so that you don't run afowl of anti-spam laws?

(4) Did the A/B test include a "No thanks, I want to read the article instead" instead of just a "No thanks" button? How much does that affect the number of sign-ups?

(5) How many of those users will actually open their newsletters vs. getting lost in the clutter of other newsletters that they automatically sign-up for?

I've run A/B tests on this and seen meaningful higher conversion rates.

To answer your questions from my experience (dealing with a non-tech audience):

> (1) How many of these users just mark all of the newsletters as spam?

The deliverability and negative feedback for users who come in through the popover is not statistically different from users who sign up in other positions (bottom of the page, for example).

> (2) How many users don't realize that they can dismiss the dialog, and enter their email because they think that they are forced to?

We did some basic user testing and the vast majority of people had no difficulty with finding the close dialog. If this were the reason, I think it would be born out in (5).

> (3) Are you only making this distinction in the smallest way possible so that you don't run afowl of anti-spam laws?

IANAL, but I don't think there's actually any law preventing an email wall. We even tested one at some point but it did have a meaningful effect on bounce rate, and it did tend to get more undeliverable emails than the dismissable version.

> (4) Did the A/B test include a "No thanks, I want to read the article instead" instead of just a "No thanks" button? How much does that affect the number of sign-ups?

Why would a business do this? What business goal would including the additional copy fulfill?

> (5) How many of those users will actually open their newsletters vs. getting lost in the clutter of other newsletters that they automatically sign-up for?

In my experience, ~20% open the emails and then 40-50% click a link within the email. These rates were not statistically different for people who subscribed through popovers vs. less intrusive forms.

That being said, I still personally dislike popovers and never subscribe through them.

> In my experience, ~20% open the emails and then 40-50% click a link within the email. These rates were not statistically different for people who subscribed through popovers vs. less intrusive forms.

That is, by far, the most ridiculous click through rate I have ever seen for a newsletter.

I get those kind of rates on welcome e-mails and product specific follow-up e-mails, but not for general blasts. That said, there are some segments of super fans that will consistently click through at a 40% rate (especially in Canada for whatever reason).

> That is, by far, the most ridiculous click through rate I have ever seen for a newsletter.

We have a pretty great fanbase. :)

I recently received an email from someone because because it was too difficult to sign up for my newsletter. Mailchimp uses double-opt in and the steps ended up being 1) fill out email 2) confirm email address 3) click 'I am not a robot' button.

My newsletter open rates are around ~45% and click rates are ~15% which both seem to be pretty good and I attribute this to the fact that you have to really want to subscribe in order to do so. Given that I pay based on the number of subscribers I have, and that the purpose of my emails is to point people to the website, I actually periodically unsubscribe people that never click/open them.

As far as overall traffic not falling off I can understand that. I close them so fast I forget which sites use them and so it never enters into the decision making process (will I go back) for me. I do find them annoying and have never signed up for one. I'm MUCH more likely to signup for your newsletter or follow you on twitter if you put links/forms to do so AT THE END of the article.

This is what I've done with my newsletter form and it works pretty well. It would likely be possible to get some more signups by adding a popover form but I'd be doing it at the risk of annoying readers that would have never signed up.

I have noticed that some sites make the popover happen when you mouse off of the window to the back button, which kind of makes sense because you're leaving anyway.

That's largely why I don't think it harms many people. It's a bit of short-term rage by those who care, many people close them out without a second thought, and even more people actually subscribe than a form in the side nav or end of the article.

> I'm MUCH more likely to signup for your newsletter or follow you on twitter if you put links/forms to do so AT THE END of the article.

Unfortunately you're in the minority. Every A/B test I've seen shows popovers performing much better than end-of-article placements on all dimensions.

Reply on news.ycombinator.com