Hacker News Reader

As the answer would seem to be 'no' (crawlers can ignore robots.txt rules), I'm wondering if GPT is going to usher in an era where new content is simply not made available to view/crawl on the web. E.g. Want to know all the cool events happening this weekend in NYC? Ask our vertical gpt-chat site to find out.

View on HN

> I'm wondering if GPT is going to usher in an era where new content is simply not made available to view/crawl on the web.

I've stopped adding new stuff to the public areas of my websites. Not as a result of GPT specifically, but as a result of a dramatic increase in the amount of scraping being done to support things that I don't approve of. AI training in general being one of them.

OpenAI uses the common crawl dataset and I would expect them to respect the robots.txt rules.

Reply on news.ycombinator.com