Have you tried using llms.txt when available? Very useful resource