Why don't we make a tool that solves poverty by taxing the rich?
The problem is that of end users, and the tool is an attempt to help them with their problem. It does require cooperation with site owners, yes, but when a site exists to help the end user...
> Why don't we make a tool that solves poverty by taxing the rich?
Well, for one, there is not nearly enough utilized resources in the world to solve poverty. Taxing everything we can get our hands on would still only provide a fraction of what would be needed to solve poverty. As things sit today, it is mathematically impossible to solve poverty.
There is all kinds of unutilized resources, namely human capital, that could potentially see an end to poverty if fully utilized, but you will never tax your way into utilizing unutilzed resources. A tool to unlock those resources would be useful, and, indeed, there are efforts underway to try and develop those tools, but we don't yet have the technology. It turns out developing such a tool is way harder than casually proposing that we agree to name a file `llms.txt`.
Or that the crib of software (california) with elite engineers (openai comp averages 900k/yr) needs help with a task that indians can do for 3 bucks an hour (web scraping)
1. There is no such assumption. Not even a mention.
2. Such an assumption would have no relevance here anyway.
Do you always struggle to read, or are you playing dumb for comedic effect?
1. Creating a tool based on helping llms train on website implies that: llms have a problem with training on websites (even though html is designed for easy machine parsing of content) and second that llms are still crawling and have not moved on to other harder sources of data.
2. I am challenging those raison d'etre assumptions on the tool. Questioning not only the tool and its usefulness, but its creator's understanding of the state of llm development.
What are you talking about? The tool has basically nothing to do with websites, other than it is assumed the author of the document will provide it to the user via their website and that the user will know to find it there. Technically speaking, the user could, instead, request the document from the author over email, fax, or even a letter delivered by hand. But HTTP is more convenient for a number of reasons.
> llms have a problem with training on websites
If you mean LLMs have a problem with keeping up with current events, yes, that is essentially the problem this is intended to solve. It offers a document you can inject into your prompt (think RAG) that provides current information that an LLM is probably not up-to-date with – that it can use to gain knowledge about information that may not have even existed a minute ago.
You could go to the regular HTML website and copy/paste the content out of page after page after page to much the same effect, but consolidating it all into one place, with an added bonus of being without any extraneous information that might eat up tokens, to copy/paste once makes it easier for the user.
> Questioning not only the tool and its usefulness
Its usefulness is worth questioning. It very well may not be useful, and the author who proposed this even admits it may not be useful – putting it out there merely to test the waters to see if anyone finds it to be. But your questions are a long way away from being relevant to the tool and how it might potentially be useful.
"The problem this solves is that today, constructing the right context for LLMs based on a website is ambiguous — do you:
1. Crawl the sitemap and include every page, trying to automatically format into an LLM-friendly form?
2. Selectively include external links in addition to the sitemap?
3. For specific domains like software documentation should you also try to include all the source code?
Site authors know best, and can provide a list of content that an LLM should use."
(There's quite a bit more info there that answers this question in more detail.)
It is extra burden for content authors to start thinking about LLM training requirements especially if those may change at a fast pace.
It is also something LLM scrapers would need to validate/check/reformat anyway to protect from errors/trolling/poisoning of the data since even if most authors would provide curated info, not all will.
You're clearly looking at this from the incorrect point of view. Silly human. Think like a bot. --The bot makers