HNHacker News
TopNewBestAskShowJobs

mzmcbride

3 karma · joined July 16, 2019

submissionscomments
mzmcbride··on Wikipedia's robots.txt
This URL was returning "Not Found" for me as well, but with an HTTP 501 status instead of the traditional HTTP 404. After investigating, the issue seems to be cookie-related:

  $ curl -sIXGET 'https://www.reddit.com/robots.txt' -H 'cookie: rseor3=true;' | head -1
  HTTP/2 501
Compare with:

  $ curl -sIXGET 'https://www.reddit.com/robots.txt' | head -1
  HTTP/2 200