Hacker News Reader

Comment by zkid18 | Hacker News Reader

What's wrong with AI agents accessing website content? We seem to have been happy with Google doing that for ages in exchange for displaying the website in search results.

View on HN

The website owner chooses. They can say "nope" in robots.txt. Not everyone respects this, but Google does. Google can choose not to show that site as a result, if they want to.

This adds a third option besides yes and no, which is "here's my price". Also, because cloudflare is involved, bots that just ignore a "nope" might find their lives a bit harder.

Robots.txt is for crawlers. It's explicitly not meant to say one-off requests from user agents can't access the site, because that would break the open web.

Yep, there's really two parts to this.

* Some company's crawler they're planning to use for AI training data.

* User agents that make web requests on behalf of a person.

Blocking the second one because the user's preferred browser is ChatGPT isn't really in keeping with the hacker spirit. The client shouldn't matter, I would hope that the web is made to be consumed by more than just Chrome.

The thing people have been doing for ages is a trade: I let you scrape me and in return you send me relevant traffic. The new choice isn't about a trade, so it's different.

And AI agents scrape your content in exchange for what exactly?

Sorry, I distinguish here an AI agent that basically automate the visual lookup and scraping to feed into LLMs by big tech. I don't see any problem with the first one tbh.

Yeah, there's a lot of confusion between AI training and AI agent access, and it's dangerous.

Training embeds the data into the model and has copyright implications that aren't yet fully resolved. But an AI agent using a website to do something for a user is not substantially different than any other application doing the same. Why does it matter to you, the company, if I use a local LLaMA to process your website vs an algorithm I wrote by hand? And if there is no difference, are we really comfortable saying that website owners get a say in what kinds of algorithms a user can run to preprocess their content?

> But an AI agent using a website to do something for a user is not substantially different than any other application doing the same.

If the website is ad-supported then it is substantially different - one produces ad impressions and the other doesn't. Adblocking isn't unique to AI agents of course but I can see why site owners wouldn't want to normalize a new means of accessing their content which will inherently never give them any revenue in return.

I don't believe that companies have the right to say that my user agent must run their ads. They can politely request that it does and I can tell my agent whether to show them or not.

True, but by the same measure your user agent can politely request a webpage and the server has the right to say 403 Forbidden. Nobody is required to play by the other parties rules here.

Exactly. The trouble is that companies want the benefits of being on the open web without the trade-offs. They're more than welcome to turn me down entirely, but they don't do that because that would have undesirable knock-on effects. So instead they try to make it sound like I have a moral obligation to render their ads.

For traditional search indexing the interests of the aggregator and the content creator were aligned. AIs on the other hand are adversarial to the interest of content creators, a sufficiently advanced AI can replace the creator of the content it was trained on.

We're talking in this subthread about an AI agent accessing content, not training a model on content.

Training has copyright implications that are working their way through courts. AI agent access cannot be banned without fundamentally breaking the User Agent model of the web.

Ok, fine, let's restrict it to AI agents only, without training. It's still an adversarial relationship with the content creator. When you take an AI agent an ask it "find me the best italian restaurant in city xyz" it scans all the restaurant review sites and gives you back a recommendation. The content creator bears all the burden of creating and hosting the content and reaps non of the reward as the AI agent has now inserted itself as a middleman.

The above is also a much clearer / more obvious case of copyright infringement than AI training.

> AI agent access cannot be banned without fundamentally breaking the User Agent model of the web.

This is a non-sequitur but yes you are right, everything in the future will be behind a login screen and search engines will die.

> reaps non of the reward

Just to be clear what we're talking about: the reward in question is advertising dollars earned by manipulating people's attention for profit, right?

I frankly don't think that people have the right to that as a business model and would be more than happy to see AI agents kill off that kind of "free" content.

> The content creator bears all the burden of creating and hosting the content and reaps non of the reward as the AI agent has now inserted itself as a middleman.

As a user agent my god what's happened to our industry. Locking the web to known client which are sufficiently not the user's agent betrays everything the web is for.

Do you really hate AI so much that you'll give up everything you believe in to see it hurt?

Like I said in another comment, I'm pointing out what is going to actually happen based on incentives, not what I want to happen. I'd much rather the open web continue to exist and I think AI will be a beneficial thing for humanity.

edit: to be clear, it's already happening. Blogs are moving to substack, twitter blocks crawling, reddit is going the same way in blocking all crawlers except google.

To be optimistic, as long as anonymous access is a thing, or creating free accounts is a thing, such crawler blocks can probably be bypassed. I hope so, at least.

Reply on news.ycombinator.com