Comment by phillipcarter | Hacker News Reader

Comment by phillipcarter | Hacker News Reader

Something I've been noodling on for a bit is that adding RAG - which should be a pattern most applications do - increases accuracy and reliability of an LLM while also making the whole system harder to understand and wrangle without good diagnostic data.

Without RAG, you're really just iterating on a prompt that you might parameterize with a few pieces of data. When you see user activity or get a report that something isn't working right, you can iterate fairly easily to get a desired outcome.

When you add RAG, now the behavior of the whole system is dependent on decisions you make upstream. Sure, if that's just one single call to dot(a,b) on a set of vectors, that's easy enough. But what if your RAG system involves calling out to several different databases? What if it pulls in other components that are specific to a user and might be unique per-request, all in the name of improving accuracy? Little doubt that it will improve accuracy, but now you've got a more complex, perhaps very distributed system that you can't just pull down locally and start reproducing behavior with.

This is why I really do believe that a lot more developers are going to have to embrace tracing as a means to understand their applications. My understanding is that LlamaIndex already has a tracing model of sorts internally. Being able to get that data out as OTLP and correlated with the rest of an application would be rad.

You are absolutely right. I have a built a handful of complex interconnected LLM calls, and the thing I am dying for is a simple way to inspect what is happening at each step. For instance, one chain has 5+ steps where data is fetched, transformed, sent to an LLM, and transformed again. I have a homegrown solution to seeing what's going on, but it is missing a lot of what I would want to see.

I've seen some attempts at providing RAG tracing as a service, but for my specific use case, where in some instances I am making 100+ LLM calls per chain, it isn't a fit quite yet.

What we do, admittedly not with chaining, is use OTel tracing end-to-end. Our RAG pipeline is (currently) 42 spans, and the end-to-end of RAG + LLM call + parse/validate is 48 spans.

That's just a part of the whole trace, though. Our feature is a natural language querying system built onto a querying engine. And so when someone has their input typed out and clicks the button (or hits enter), we trace that user interaction, RAG, LLM call, parse/validate, and queuing up to run against the querying engine. Getting that "whole experience" visibility really helped us iterate in production.

All of it was done just by manually instrumenting things with opentelemetry tracing and having our own conventions for naming key:value pairs that we're looking to upstream. It's more work to set up if you're not familiar with opentelemetry, but it's worth it because you can then start plugging in user IDs or really any other interesting data that helps correlate with LLM behavior so you get a full picture of what's going on.

> increases accuracy and reliability of an LLM while also making the whole system harder to understand

Harder than what? If the alternative is a black box fine tuned model, RAG is obviously superior. There’s no easy way to audit why an LLM spits out nonsense, and the only way we know how to make it less nonsensical is to feed reasonable inputs.

So RAG replaces a poorly understood problem (zero-shot generation from decoder-only models) with a well-understood one and a poorly understood one (search retrieval followed by zero shot generation from documents).

I think it’s a little disingenuous for all of these out of the box repos to tout their GitHub stars and claim to be “leading” in RAG. The first step is retrieval. People have been working on retrieval for decades and vector-based retrieval since before attention was all we needed.

That said, I’m glad that the prospect of text generation has forced people to become familiar with information retrieval. It’s a relatively new subject compared to the rest of CS and I really only learned about in industry.

Theres definitely going to be a lot of startups in and around this space.

I've been building a basic RAG system DIY and just built a basic admin panel that displays these traces by dumping a lot of data into a postgres table. But this type of observability, configurability, and ability to score results and get feedback loops, version, etc etc will be really key.

I'll just add that there are use cases where the extra layer actually adds a nice element of control and visibility. Adjusting a prompt to a black box LLM can sometimes lead to unpredictable results, though that can be mitigated with a good test set to some extent.

Reply on news.ycombinator.com