Hacker News Reader

Apparently, the regular search crawler does it, but the ai thingie doesn't.

Can confirm. My website is flooded with AI bots despite attempts to block crawlers to certain parts of it.

Huh? You can add Google-Extended[1] to opt out from Generative AI summaries.

[1] https://blog.google/technology/ai/an-update-on-web-publisher...

Google will still scrape it for training data either way, this only impacts search results.

> Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help *improve Bard and Vertex AI generative APIs*, including future generations of models that power those products.

https://www.theverge.com/news/630079/openai-google-copyright...

they're literally asking to break laws to train AI for national security. A sentence in a press release from 2 years ago is worthless... look at what they're actually doing

Reply on news.ycombinator.com