Hacker News Reader

Comment by caseyy | Hacker News Reader

Yes! Please standardize the web into simple hypertext so “LLMs can use it”. I promise I won’t build any tools to read it without the ads, tracking, and JavaScript client side garbage myself. I will not partake in any such efforts to surf the web as it was intended be, before its commercialization and commodification. No, sir, I could never!

View on HN

The problem I see is that people will develop two versions of websites -- the LLM version, optimized to give the model a good impression of their products or services and get them into the training data, and the human version (with sub-versions for mobile and such) which will be SEOd to hell.

No one wins in the long run by creating technical solutions to human incentive problems. It is just a prolonged arms race until

* the incentives are removed

* the process is made so technically complex or expensive that only a few players can profit from them

* it is regulated such that people can make money doing other things which have better risk/reward

* most people just avoid the whole ecosystem because it becomes a cesspool

A brief guide on avoiding this pitfall, for search engine and LLM operators:

Step 1: Punish ranking and visibility of sites whose llms.txt differs from a random sampling of actual web (HTML) content.

Step 2: There is no step 2.

If you can reliably machine-diff format A and format B, there's no need for two different formats in the first place.

Right. And so you'll end up punishing some sites that do it perfectly, while other sites that game the algorithm game first priority.

> No one wins in the long run by creating technical solutions to human incentive problems. It is just a prolonged arms race

But maybe this time the naive technical solution will work

Everything is temporary.

You, me, LLMs, Google, humanity, Earth, Sol...

We can choose to carry on and perform [what we think are] improvements and make the best of it, or we can choose to cash it in early and simply give up.

Brief guide for site optimizers:

1. Figure out how to embed content that only LLMs see which affect their output

2. Wait for that to stop working

3. Innovate another way to get past new technical problem

I wish this were true too but it doesn't appear that llms.txt is meant to be a markdown version of the site's content based on the example on the site and after peeking at a few llms.txt files on some of the sites in the directory.

It looks like its text meant to be fed into the llm as a system prompt specific to the site.

The most simple ones just look like a sitemap restricted to documentation: https://www.activepieces.com/docs/llms.txt

Some interesting stuff is in some of them. Like this one that prompts the LLM to explain to the user the ethical issues of using AI agents along with a disclaimer:

https://boehs.org/llms.txt

That site appears to be someone's blog and they don't seem like big fans of LLMs.

https://boehs.org/node/llms-destroying-internet

A pretty clever use of llms.txt.

Reply on news.ycombinator.com