Hacker News Reader

Comment by lukev | Hacker News Reader

Yes, a bit, though an important feature here is it's still searching the underlying data sources (e.g. Google, Wikipedia, or others) and then using a LLM to summarize the results.

The "natural language tokenizer" itself is often an LLM (they do a pretty good job of this).

A further extension this article doesn't talk about is to have a LLM with a different prompt analyze the answer before returning to the user, and do more queries if it doesn't believe the question has been well answered (imagine clicking "next page" of google search results under the hood).

The potential complexity of this scales all the way up to a full "research assistant" LLM "agent" that calls itself recursively.

View on HN

Before learning about RAG I thought that it is recurrent LLM agent that traverse over documents. After some study I must say that VectorDBs are boring.

It can be as simple or as complicated as you want. The article starts off by saying the naive approach of just embedding the query and looking for similar documents is a bad approach and what you actually want to embed and compare is something similar to the expected result. They don't go into detail on this but using their example of "what is the capital of France" you would conceivably transform that into "list of European capital cities" or "list of cities in France" using an LLM, embed that, find the similar documents, feed those documents into an LLM along with the query and some system instructions about how to format the response and then return that. Keep in mind this is an absurdly simplified example query and none of this process is needed to answer the actual question which the LLM would know from its training data but you would want this process in place to ensure accurate results for more complex or specialized queries.

> none of this process is needed to answer the actual question which the LLM would know from its training data.

I think this isn't true; even if the model has the answer stored implicitly in its weights, it has no way of "citing it's source" or demonstrating that the answer is correct.

if your model can't predict the completion of "the capital of France is _" then it's going to really suck for other completions

This is a great example of something GPT-4 gets confidently wrong, today. I just ran this query:

Prompt: "The year is 894 AD. The capital of France is: Response: "In 894 AD, the capital of France was Paris."

This is incorrect. According to Wikipedia, "In the 10th century Paris was a provincial cathedral city of little political or economic significance..."

The problem is that there's no good way to tell from this interaction whether it's true or false, because the mechanism that GPT-4 uses to return an answer is the same whether it's correct or incorrect.

Unless you already know the answer, the only way to be confident that a LLM is answering correctly is to use RAG to find a citation.

lol you just gamed it with an edge case where the most likely completion is incorrect and you're proving my point that the simple case doesn't need RAG but weird complex edge cases do.

But how is the user supposed to know if their query is a weird complex edge case or not? The model certainly can't tell them.

which is why you would use RAG if you want your app to be an encyclopedia

What, because you want your encyclopedia to be confidently wrong on everything that -- unbeknownst to both you and your encyclopedia reader(s?) -- happens to be "a complicated edge case"? (And what isn't "a complicated edge case", in some way or other?)

Eh... Does that really make sense to you?

Reply on news.ycombinator.com