HNHacker News
TopNewBestAskShowJobs

newpeak

1 karma · joined April 2, 2024

newpeak@163.com
submissionscomments
newpeak··on Dense Vector and Sparse Vector and Fulltext and Tensor Reranker = Best for RAG?
What's your advantages over paradedb? it also has dense+sparse+bm25
newpeak··on Ask HN: RAG and unstructured data from several docs
Tagging each paragraph is not a good approach. Instead, using LLM to generate a summary based on the clustering of paragraphs could be a good alternative. That's what RAPTOR(https://arxiv.org/html/2401.18059v1) has suggested.

Regarding to reranker, it's a pure dynamic solution, which is different with re-chunking which requires all data to be reindexed.

Comparing two paragraphs, in most cases, it's based on embedding, which means each paragraph will correspond to a single embedding. So comparing them does not take words into account. However, if you adopts a hybrid search which will use full text search as another kind of recall approach, it will take the words into ranking consideration, in that case, the scores are computed based on the TF/IDF metrics within the paragraph, which is an accumulated score of all hitted tokens.