RAG is a hack
latent.space
latent.space
Without RAG, you're really just iterating on a prompt that you might parameterize with a few pieces of data. When you see user activity or get a report that something isn't working right, you can iterate fairly easily to get a desired outcome.
When you add RAG, now the behavior of the whole system is dependent on decisions you make upstream. Sure, if that's just one single call to dot(a,b) on a set of vectors, that's easy enough. But what if your RAG system involves calling out to several different databases? What if it pulls in other components that are specific to a user and might be unique per-request, all in the name of improving accuracy? Little doubt that it will improve accuracy, but now you've got a more complex, perhaps very distributed system that you can't just pull down locally and start reproducing behavior with.
This is why I really do believe that a lot more developers are going to have to embrace tracing as a means to understand their applications. My understanding is that LlamaIndex already has a tracing model of sorts internally. Being able to get that data out as OTLP and correlated with the rest of an application would be rad.
I've seen some attempts at providing RAG tracing as a service, but for my specific use case, where in some instances I am making 100+ LLM calls per chain, it isn't a fit quite yet.
That's just a part of the whole trace, though. Our feature is a natural language querying system built onto a querying engine. And so when someone has their input typed out and clicks the button (or hits enter), we trace that user interaction, RAG, LLM call, parse/validate, and queuing up to run against the querying engine. Getting that "whole experience" visibility really helped us iterate in production.
All of it was done just by manually instrumenting things with opentelemetry tracing and having our own conventions for naming key:value pairs that we're looking to upstream. It's more work to set up if you're not familiar with opentelemetry, but it's worth it because you can then start plugging in user IDs or really any other interesting data that helps correlate with LLM behavior so you get a full picture of what's going on.
Harder than what? If the alternative is a black box fine tuned model, RAG is obviously superior. There’s no easy way to audit why an LLM spits out nonsense, and the only way we know how to make it less nonsensical is to feed reasonable inputs.
So RAG replaces a poorly understood problem (zero-shot generation from decoder-only models) with a well-understood one and a poorly understood one (search retrieval followed by zero shot generation from documents).
I think it’s a little disingenuous for all of these out of the box repos to tout their GitHub stars and claim to be “leading” in RAG. The first step is retrieval. People have been working on retrieval for decades and vector-based retrieval since before attention was all we needed.
That said, I’m glad that the prospect of text generation has forced people to become familiar with information retrieval. It’s a relatively new subject compared to the rest of CS and I really only learned about in industry.
I've been building a basic RAG system DIY and just built a basic admin panel that displays these traces by dumping a lot of data into a postgres table. But this type of observability, configurability, and ability to score results and get feedback loops, version, etc etc will be really key.
Anyway, Jerry and LlamaIndex has been a huge source of learning and inspiration—please keep tweeting and publishing, Jerry!
But better answer: "retrieval augmented" has been used in papers since at least 2018 [1], the full "RAG" acronym since at least 2020 [2], and it was recently repopularized when the GPT3+ hype wave realized they were doing the same thing.
[1] https://arxiv.org/abs/1808.03430 [2] https://arxiv.org/abs/2005.11401
But all it really means is that you put some extra information in your prompt (from somewhere) that didn't come from the user. It's basically the first thing that anyone thinks of once they get the API call working.
People don't use it to only mean vector search, despite often assuming that is what it means.
What I have been leaning towards recently is not using vectors at all, or only using it when I have an extremely close vector match. Instead I give the API one or more functions to look up information that is like an index for documentation or retrieving some info from a database. But it's specific to the application.
I think usually the non-vector stuff is more effective.
Yes, and if you are getting (retrieving) some extra information to add to (augment) your prompt for further responses (generation) then it's not very ambiguous.
I don't think that's correct, but if you look in this discussion, that's the reality of usage. So that makes it ambiguous in practice.
There are situations where you have some limited amount of relatively static data you want to add to your model. In those situations you'll get value from fine tuning the model.
But if you have a large set of data to add, or data that updates frequently, then a model that understands human language and basic logical reasoning hooked into a RAG makes much more sense. That strikes me as an elegant solution. I don't see that as "a hack" at all.
The thing I love about talking to all the founders we've chatted with so far is their intellectual honesty and pursuit of solving real problems that people have, not just creating ones out of thin air. RAG -is- a hack, but its a very good hack, and findings like the Lost in the Middle paper and even learnings from finetuning so far make it such that RAG frameworks are here to stay.
I was going to post https://www.latent.space/p/llamaindex#transcription as a shortcut to it but apparently that just confuses the on-page JS and it resets to #details instead.
Admittedly, yours is still in the "left open in a tab" state, but that's because I'm not particularly familiar with the subject matter and so need a period when I'm awake, not doing anything else, and can actually focus on it and google things if necessary rather than just getting lost.
I disagree with that. The model isn't fetching information to augment its responses, nor does it have any notion of ground truth or accuracy. I would argue that RAG points at the inverse - we have these incredible probabilistic autocomplete machines but they're so incapable of being reliable that we have to force them to work in a context that we establish for them.
How does it decide which information to request? How does it know which is the right or wrong information for a given request? Maybe it's possible to wave this away with "something in the future will do that!" but that's literally the entire reason why all current AI is limited. It doesn't have the business context or reasoning capabilities to make the right decisions to decide what the right data to augment generation even is. I don't believe we're any closer to AGI.
I think LLMs are fairly good at deciding what information is relevant to a query or not though, it is rare that I have found an LLM to be distracted by RAG results (even when they're not relevant)
So have it be a separate module? The human brains is compartmentalized too.
[citation needed]
Obviously, the human brain continuously integrates new material, but whether that's more like RAG, continuous fine-tuning, just having a really big context window, or just having a completely different model architecture where activiation (or not) of pathways in inference also produces weight adjustments directly — or doesn't meaningfully work like any of those — is another question, where the answer is less obvious.
I think RAG (storing exact data losslessly with some sort of indexing to allow retrieval, and something hidden-prompt-like directing the use of that in populating an inference context) is a lot less likely that something that involves dynamically adjusting model weights. Which also explains forgetting, not just learning.
I see these two claims as orthogonal. AGI need not work like the human brain, and I’m not sure people’s brains have any lossless representation of memories like a RAG system would. People are infamously unreliable when it comes to recalling information during crimes for example.
However, it does seem like a RAG system is much more versatile than continuously training. Facts change all the time and it’s infeasible to retrain every time that happens.
Great episode!