Making My Blog Readable by AI Agents

deallocatedobjects.com

37 points·ingve··10 comments

One trap when you name AI bots specifically is that is that robots.txt groups don’t merge. Crawlers will only follow the most specific group so if GPT bot has its own group named nothing under User-agent: * applies to it anymore so you need to repeat every disallowed line under each group.

Are llms.txt actually read by bots now? I found one experiment that showed that no bot has accessed the file in 3 months.

https://www.rebootonline.com/geo/llms-txt-experiment/

I get the odd one try on my personal site, but usually its just robots.txt... /shrug

I once had oh-my-pi read a website’s llms.txt while doing research many months ago.

Most of what the author is doing won't actually help.

> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.

https://www.longato.ch/llm-md-files/

> Across all 50 sites, across 200+ days, across every major AI user-agent logged - the `/llms.txt` endpoint was requested zero times.

https://mishamanko.com/research/what-ai-bots-actually-read

Maybe read how webfetch and websearch works inside the harnesses.

why? You're helping AI companies to take your writing for free

Whilst I agree with most criticism of AI/LLMs but most of the time at the other end of someone using Claude/ChatGPT there is still a human there, a human that is trying to solve a problem or learn something. Maybe I'm naive and it's just all bots scraping sites to build their corpus of data, but at least the way I use AI is to solve problems and build things I couldn't do before (either because of time, or it was out of my reach)

Sooo, are they reading the md?

https://agents.md/

Reply on news.ycombinator.com