Hacker News Reader

Parent Full thread

scandinavian

The URL in the search is this: https://shoprocket.io/email-confirmation/34b35b1...

I don't see that in the robots.txt https://shoprocket.io/robots.txt

User-agent: *

Disallow: /cdn-cgi/l/email-protection

Disallow: /login

Disallow: /register

Disallow: /404

Am I missing something?

View on HN

You may be seeing a stale version, try this: https://shoprocket.io/robots.txt?bypass=1

(I made a lot of changes today when testing all, including "visit as Bingbot" from their webmaster tools with and without the URL blocked by robots.txt)

Reply on news.ycombinator.com