HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by warrenm | Hacker News Reader
Parent
Full thread
warrenm
·
only if the scraper/crawler actually
pays attention
to robots.txt
View on HN
dutchbrit
·
Which they ultimately should do when crawling a site. OpenAI does according to ChatGPT.
warrenm
·
as a coworker of mine liked to say, "'should' is a funny word"
Reply on news.ycombinator.com