Hacker News Reader

Comment by chaz | Hacker News Reader

r/The_Donald is "quarantined" by Reddit, which means you get a gateway page if you visit for the first time, asking if you really want to visit it (try an incognito window). In the HTML, you'll see that there's a "noindex" metatag that Reddit is instructing Google not to index it.

Also, np.reddit.com seems to be excluded from all search engines: https://np.reddit.com/robots.txt.

I'm not sure why Bing is ignoring both the robots.txt and the "noindex" metatag.

View on HN

In another comment [1] I compare this with a pro-Marxist subreddit which is similarly "quarantined" and has noindex.

Google indexes it perfectly. [2]

[1] https://news.ycombinator.com/item?id=21316065

[2] https://www.google.com/search?q=Reddit+ChapoTrapHouse

I understand why one might expect r/The_Donald and r/ChapoTrapHouse to have similar results, since they're both politically extreme subreddits. But after having worked in SEO for a pretty long time, I can't conclude that Googlebot is evaluating political affiliation with n=2 examples on n=1 sites. Instead, incoming links quality/quantity, popularity, competition, newsworthiness, age, and dozens of other ordinary differences between these pages and queries are usually the reason. Even the underscore in The_Donald likely matters.

It can get pretty frustrating to rank #1 or #2 for a query, and not rank at all for a slightly different, higher-traffic query, like using the plural, a hyphen, different capitalization, or an article.

It could be that Bings index is outdated.

Reply on news.ycombinator.com