Typically, you wouldn't manually select the documents in a query, you’d use this as part of a system wrapped around the LLM where a query (possibly itself from the model like other tool invocations) would be sent out to a web api, vectord DB, etc., return documents that would be fed back in to the model as part of a continuation query, and then the model would frame a response citing the relevant sources for elements of the response.
Patently untrue. If I have hundreds of PDF research papers totaling thousands of pages - just because I have them doesn't necessarily mean I know where to search. Having an LLM be able to find related pieces across all of these documents and extrapolate is tremendously useful.
A thousand pages is something reviewable by hand (albeit slowly) and easily amendable to grep. Anything that trivially fits in computer memory doesn't benefit from approximate search methods. I stand by what I said.
"Completely useless" might be too strong. In a context where you're already doing RAG it would help you verify what's produced.
Certainly it's radically less useful than if it could produce citations to the training set, though.
Sounds to me like you want a search engine, not an LLM.