enterprise GTM has its own set of challenges and needs and warrants someone really focused on it
515 karma · joined December 10, 2010
YC Badge: 0x9F755062601B9bBA3354c090e11A13a508d7CA0e
We noticed that most people make their decisions based on popular benchmarks scores. However, widely used benchmarks like MTEB are often overly clean, generic, and in many cases, have been memorized by the embedding models during training. To address this, we introduce representative generative benchmarking—custom evaluation sets built from your own data and reflective of the queries users actually make in production.
We just published our in-depth technical report on this, and you can run a custom benchmark locally with the Chroma CLI.
I didn't see this in the write-up, so adding it here as a common foot gun.
With the exception of Pinecone: Chroma, Qdrant, Weaviate, Elastic, Mongo, and many others store the chunk/document alongside the embedding.
This is intentional misinformation.
a distributed database is naturally has more complexity, but we’ve put a lot of effort in to make it as easy as possible to run.
Yes - that's a great explanation.
Relatedly ColPali (https://arxiv.org/abs/2407.01449) is gaining a ton of steam in the IR community.
In general we are skeptical of silver bullets in retrieval.
if you are building locally and dont want to send your data anywhere - try the open-source alternative Chroma https://github.com/chroma-core/chroma
be cautious with pgvector - the recall can be extremely bad (% retrieved nearest neighbors vs ground truth)
> It would be wonderful if there were a simpler (single file, SQLite or DuckDB like) database for vectors than the complex (and in some cases, unfortunately cloud-based) ones available now.
This is literally chroma!
All you have to do is chunk... I'd start at a paragraph and experiment.
A Ruby client would be great. Our FastAPI spec makes this pretty easy - it's at localhost:8000/openapi.json when the docker backend is running.