HN Banned from Google
linkaudit.co.uk
linkaudit.co.uk
Specifically https://news.ycombinator.com/item?id=5955043 from last week, complete with a response from Matt Cutts (https://news.ycombinator.com/item?id=5955374):
> It's not that PG has a grudge against Google (or vice versa) or anything like that. I believe that search engine bots crawl Hacker News hard enough that PG blocks most crawling by bots. In the case of Google, he does allow us to crawl from some IP addresses, but it's true that Google isn't able to crawl/index every page on Hacker News.
> Here's a link where I answered the same question about three weeks ago: https://news.ycombinator.com/item?id=5837004 , so this isn't a new issue. In fact, PG has been blocking various bots since 2011 or so; https://news.ycombinator.com/item?id=3277661 is one of the original discussions about this.
> And to show this isn't a Google-specific issue, note that Bing's #1 result for the search [hacker news] is a completely different site, thehackernews.com: http://www.bing.com/search?q=hacker+news
> In general, I think PG's priority is to have a useful, interesting site for hackers. That takes precedence and is the reason why I believe PG blocks most bots: so that crawling doesn't overload the site.
Google doesn't want other people (who are unaware google is selling their privacy to the NSA) to accidentally stumble across the comments on HN.
Btw this is showing the pages indexed for the site HN. That its not showing in the results now for your brand WILL have devastating results on your traffic directly from Google. Though you might not care?
http://ycombinator.com/newswelcome.html
This is arguably the best result for the query, since it explains what HN is about and provides a link to the front page. (People who have already used HN know where it is and don't need to search for it.)
I block cookies from Google, so they're not relying on my search history to give me relevant results.
And I'm not sure why searching for "HN" would be reasonably expected to get someone to Hacker News. After all, "hn" is the top-level domain for Honduras, and could stand for a million other things as well. It's probably accidental that it ever worked at all.