Hacker News Reader

Comment by alchemist1e9 | Hacker News Reader

Do you have any insights to share around chunking and labeling strategies around ingesting and embeddings? Qdrant I remember had some interesting abilities to tag vectors with extra information. So to be more specific the issues I see are context aware paragraph chunking and keyword or entity extraction. How do you see this general issue?

View on HN

This is the hardest part to get right (if it can be gotten 'right'), and the thing I'm always curious about.

All of these RAG solutions are implementing it with vector search, but I'm not sure it's the be all end all solution for this.

Hybrid search is definitely worth exploring (e.g. adding in TF-IDF). I believe there is such an implementation out of the box with Weaviate.

I have tried many techniques and seen others try many different techniques. I think the hardest part is selecting the RIGHT technique. This is why it is somewhat easy to deploy a RAG pipeline but very hard to optimize one. It's hard to understand why it's failing and the global implications of design choices you make in your ingestion / embedding process.

I’ll give you an analogy. We can imagine the content ingested is like a textbook, which would have a table of contents and an index. Now we lookup a topic, we find it in the TOC then likely we should read that chapter in whole, we find it in the index, then we likely read all the chapters it’s mentioned.

I’d suggest RAG might perform better if it worked somewhat like that, the chunks for embeddings should be paragraph and sentence aware, and ideally should be tagged with any existing TOC or natural sections/headings that exist in the document. This approach would allow a retrieval logic that provides cohesive information, like an entire chapter or at least 3 paragraphs prior and 3 after the matched vector.

Are you suggesting adding another step after retrieval, to add the surrounding context before submitting it to the LLM?

Yes absolutely. Often to understand the context of a topic it requires more than just the specific paragraph it is discussed and the LLMs are actually perfectly capable of understanding fairly complex concepts if provide the raw material properly.

This seems similar to recursive summary indexing and then a choice of which linked chunks to load upon a match

There are a lot of 'policy' choices here, so I've been curious how to automatically decide. The KG, GNN, and IR literature seems to have a lot of techniques relating to this, so been very non-obvious to me

> Qdrant I remember had some interesting abilities to tag vectors with extra information.

like what? adding metadata to vectors isn't by any means new, as other reply says that's just "hybrid search"

https://qdrant.tech/documentation/concepts/payload/

Perhaps it’s not that different than other solutions have but my impression was their payload system was more flexible and extensible than competing options, especially with the speed that Qdrant delivers, so for example pg_vector can leverage all of postgres capabilities but I don’t believe it can compete with Qdrant on vector specific problems.

qdrant vs pgvector remains to be benchmarked for a range of relevant sizes, please drop any if you see them!

the only time we've found pgvector to be preferable to qdrant is when the per-category embeddings are very few. E.g. in cases where filtering is expected to reduce the dataset greatly for each query (>99.9%).

This is when the relational nature of postgres really shines - you can even do in-mem calculations for distance.

Qdrant does this as well, with 100% accuracy. But it's not on small categories, but rather on small segments.

You could put different categories in different shards though, resulting in the same effect.

You can also specify exact=true to do the same on any size, but this can get rather slow when your collection grows.

Disclaimer: I'm part of Qdrant.

Reply on news.ycombinator.com