It was tuned for a specific set of open-source models we run ourselves on our own GPUs, so I can't share exact golden numbers (for example, if I replace those small models with Claude Haiku+Cohere Embed, the results get worse). A proper reranker helped tremendously because it removed noise; BM25 helped a lot too, because in many cases you want exact-match searches instead of fuzzy/vector search (so again, less noise).
For small open-source models (we used them because we wanted speed), prompt engineering mattered too, especially in cross-language benchmarks where the model may get confused about which language it should respond in (the system prompt's language, the user query's language, or the documents' language). What mattered was even the order of fields in the output JSON schema (in intermediate steps), because LLMs are autoregressive: if you order the fields incorrectly, the model may guess or hallucinate during extraction when the first value in the schema can't be extracted reliably without first considering other dependent fields that should've been extracted earlier (we don't use reasoning models to save on speed).
I used LLM-as-a-judge to quickly figure out what improved scores and what didn't. Then humans tested it manually too and calculated scores to see whether their scores diverged from the machine's scores. I think if I had to do it again, I probably would use an agent (like autoresearch) to autonomously find the best configuration for the exact set of models via intelligent bruteforce (dunno if it would work, but interesting to try).
We don't have 1B+ vectors; our system is split into tenants (organizations), a single tenant usually doesn't have that many vectors, plus every document in the system has a specific hierarchical structure, so your mileage may vary
https://cdn.aaai.org/ojs/20591/20591-13-24604-1-2-20220628.p...
Steps would be:
- A first hybrid query with bm25+query vector
- rank some first results locally (ask the LLM for relevance, usefulness? ask the human?)
- query in active learning manner; providing positive and negative examples to orient the direction of better results. It'd silence the polluting components from the original query
- iterate
If anyone's hiring for remote roles, I'd love to give that technique a try