Hacker News Reader

I wonder if search engines actually use rules for Googlebot when parsing some (specific?) robots.txt.

"Googlebot" might as well be the "Mozilla/5.0" of search index bot user agents.

Twitter is benefiting from network effects by now. I'm not even sure when have I encountered a tweet in search results last time.

I encounter tweets all the time when I'm trying to find out more about recent events.

Does twitter rely on that? Most of the time I go to twitter is from news stories, comments or other social media.

They have a section in the robots.txt that allows Google. They must think no other search engine matters.

Any revenue damage from this will be faulty minor compared to the damage that has already been done.