Hacker News Reader

Comment by 3steve | Hacker News Reader

Is removing content the same as disallowing it in a robots.txt? Because they have an interesting article [1] disallowed in their robots.txt that was added a while after it was published.

[1]:www.npr.org/2013/03/21/174840895/sexual-violence-victims-say-military-justice-system-is-broken

View on HN

> Is removing content the same as disallowing it in a robots.txt?

No.

> Because they have an interesting article [1] disallowed in their robots.txt that was added a while after it was published.

Oddly, it's a different seemingly benign story for me:

        Disallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade

IDK why they include a seemingly random story in their robots.txt. I don't think the intention is to hide the story, though.

I get the same link as you in there. I wonder if this is some honeypot technique for detecting misbehaving robots?

Reply on news.ycombinator.com