Hacker News Reader

This is an interesting sentiment given how desperate AI labs seem to be source any new internet content from any walled-garden platform willing to take their money (and how willing they are to try & take it even if you don't consent).

Abusive, sneaky scraping is absolutely through the roof.

View on HN

I feel as though you are confusing AI use in scraping by random companies and actual AI companies scraping. The AI companies seem to see value in walled garden sources like Reddit, Stack Overflow, etc. However, I don't think there has been any major instance of a major American AI company doing aggressive online website scraping and not respecting robot.txt.

Per https://thelibre.news/foss-infrastructure-is-under-attack-by..., all of the major American AI companies are not respecting robot.txt and participating in the AI-fueled DDoS of the internet.

The issue is that UA are editable by the user, and there is no proof that some random person/scraper isn't just using a suspected trusted bot's UA string. Every ethical service also posts what IP addresses they use, so that people can compare the traffic they get to see if it is actually their bot scraping. What this article describes is the game of every third-party unethical scraper; they do anything and everything to try and get their request through. They steal UA's, they steal residential IP addresses through botnets, they attempt to circumvent CAPTCHAs using AI, etc. So the behavior in this article is not prove for any major AI provider doing unethical scraping.

Reply on news.ycombinator.com