i think you prob need a bit of both. would you add llms.txt to your docs or make it crawl like a human?