Don’t try to index all the web, pick some smaller universe of quality content to search.
For example, if you are programming you should have some tool that looks at your project.toml or package.json or POM and looks up the exact versions of the dependencies you use and search that documentation and only that documentation.
For that matter you would want an AI to act only based on that knowledge base —- it is so awful to argue with a AI over whether you can write a switch statement with a null in it (you can in JDK 21) or have it write code for the wrong version of SQLAlchemy.