Got you. You made me realize something. I've been developing a scraping api and after reading your post I tried to do the same as you did. First I've asked claude about the company, the services and prices. Than I've used the api to test if the response is the same and It's not. Basiclly using the api I got thinks that just asking claude, didn't show. The difference is that api turn the website into markdown and don't see the llms.txt. Using api looks like the AI understand the company as a whole.
Thanks for sharing that experiment, you gave me a good ideia on creating a brain for my scenario. I'm thinking of use Obsidian as the brain, and use my scraping api to colect the data from my company website, linkedin and instagram. With all the knoledge, use paperclip to orchestrate marketing, projects, fixes.
I will build that.