Comment by CShorten | Hacker News Reader

Comment by CShorten | Hacker News Reader

Parent Full thread

1. Please expand on how you are defining full text search distinctly from sparse search to continue the discussion. My understanding is that full-text search is built on inverted indexing which is the intended meaning behind "sparse" search -- whether that be BM25 sparse or SPLADE sparse or exact keyword / phrase matching, my understanding is that it is the same underlying index algorithm with distinctions in scoring for BM25 / SPLADE.

2. The original argument references "combined system" in the sense of Hybrid Search (BM25 + Dense Vector Search). I don't think this is a fair comparison -- model inference services are extremely lightweight relative to the sparse / dense indexing systems we are primarily discussing. I also have not advocated for Cross Encoder inference in the spirit of improving latency, just clarifying why a module system is used for it.

3. The hybrid search results from researchers independent of either of us are linked in the original comment, here it is again - https://arxiv.org/abs/2201.10582. Your criticism of the Weaviate ANN benchmarks isn't relevant to our discussion on Hybrid Search. I have linked this to show that Weaviate has produced comparative benchmarks which was your original claim. Although I do not agree with your premise that a full-scan search with give similar speed results to HNSW on this setup, or however arbitrarily we are defining "good results". I acknowledge that it is not included in the benchmark report and is something that should be added. I also agree that it would be interesting to run ANN recall tests on several hardware configurations.

> Please expand on how you are defining full text search distinctly from sparse search to continue the discussion

In addition to the indexing algorithm, there is the tokenizer, which depends on the language, lemmatizer, synonyms, stop-words, and so on and so forth. In addition, the ranking function itself may be quite different and based on different rules. See how Meilisearch does it. Reducing full-text search to just a reverse index is a misconception

> Your criticism of the Weaviate ANN benchmarks isn't relevant to our discussion on Hybrid Search.

It is very much relevant, as I mentioned, in parallel processes

total_latency = max(BM25_latency, Vector_search_latency) + merge overhead

and my claim is that in specialized tools both BM25_latency and Vector_search_latency will be better than what the multi-tool system can provide.

> I have linked this to show that Weaviate has produced comparative benchmarks which was your original claim.

I don't see any comparisons in your benchmarks here - https://weaviate.io/developers/weaviate/benchmarks/ann

You just benchmarked yourself, that is not interesting and not helping.

> I also agree that it would be interesting to run ANN recall tests on several hardware configurations.

That is not the point. In our benchmark we run all engines on exactly the same machine to make it fair. Sometimes same configuration in different regions already gives very different performance on some cloud providers.

1. Weaviate does all these things as well, please see - https://weaviate.io/blog/pulling-back-the-curtains-on-text2v.... Although there are interesting optimizations around fitting a tokenizer, these are all relatively simple things whereas the inverted index is more so the interesting thing where the optimizations of specialized systems come into play e.g. WAND scoring.

2. Following on #1 what optimizations does BM25 require that justify an entirely separate tool that requires maintenance of two separate search systems? Also helps that merge overhead to have both searches in the same system.

3. Any company's report of benchmarking itself against its competitors should be taken with a grain of salt.. this is obviously bad practice. The purpose of these benchmarks are to compare in this case hyperparameters of HNSW and in future works around the BEIR numbers I provided earlier, Hybrid Search performance.

4. Again, no company can seriously benchmark itself against its competitors due to obvious conflicts of interest. Maybe this competition could be hosted here - https://big-ann-benchmarks.com/index.html#organizers.

Benchmark is open source - https://github.com/qdrant/vector-db-benchmark You are welcome to fork it and make your own measurements, if you suspect something.

Benchmarks like https://big-ann-benchmarks.com/index.html#organizers are good for comparing algorithms, but not engines. They are focused on a single use scenario and do not cover variety of possible applications. Like, for example, how filtering affect the performance.

Reply on news.ycombinator.com