20x faster than pgvector: HNSW index in Postgres with pg_embedding
neon.tech
neon.tech
- Given how neon architecture decouples compute with storage using the safekeepers & pageservers will this still just work with neon? (Wondering because you mention the index is in-memory so unless your stateless compute nodes can somehow hot-swap the indexes I wasn't sure how that'd work?)
- Do you plan to offer vector search as a plug-and-play offering? If so does neon as a product offering plan to introduce more "out-of-the-box" functionalities like vector search? (Similar to xata's offerings like search & vector search.)
- unrelated question- I believe neon has very small cold startups for free/scale-to-zero configurations but are there also inherent small latencies for infrequently accessed data? In other words is it safe to say if there were a large table with records that are "old"/"archival" but also always a sort of ~last 30-days of records that are "fresh"/"accessed with more frequency" would there more likely be slight latency introduced when accessing the older records?
Neon looks awesome and thanks for neons open source contributions!
- This does work with Neon today, please go ahead and try. Today the index is in-memory and we are working on the on-disk implementation. Neon architecture does make it trickier - no local disk, so we are actively experimenting as we speak. Anything we ship will of course work on Neon.
- We will start with vector ANN and see the reception and user feedback. Now that we have 100K databases we are super attuned to what our users tell us.
- There is a cache/storage hierarchy: local cache on compute, cache on pageserver, and S3. We are working on cold start (stay tuned for more announcements this week). The goal is to make this unnoticeable for the app.
"Qdrant currently only uses HNSW as a vector index."
- https://qdrant.tech/documentation/concepts/indexing
So it would be interesting to see benchmarks between pg_embedding and Qdrant. I would expect them to perform similarly, but perhaps there are other factors?
This is different from the dataset we used. We used GIST-960. It could interesting to test different vector store indeed.
This was a relatively quick project for us and the index is currently in-memory. However it is fast! We would love your feedback and excited to invest further.
Virtual screens of molecules frequently use steps to filter out molecules, one of which is via chemical similarity.
If we consider pg_embedding ready for use, why don't you compare with pgvector:
1) The need to rebuild index every instance restart,
2) Replication support?
The blog post does mention memory limitations, but we could elaborate more on the fact that the index is entirely in-memory and clarify the user-visible consequences of this fact. We'll edit the post for better clarity.
E.g. If the DB table had 500k vectors, but you were pre-filtering by WHERE client_id = X (returning only 200 rows) then with an AND <embedding search> (returning only 6 rows) - would this still have the same performance issue?
Or is it literally if the embedding search is over 500k rows?
Also, since this is in memory, I assume this significantly affects startup time in order to rebuild the index? Would be nice to see how bad that is for larger vector datasets.
It doesn't yet have a LICENSE file, so it's not strictly OS quite yet, but that might have been an oversight.
It would have been nice to get the support of neon in progressing pgvector - since it’s already so widely adopted by the community
(disclosure: supabase ceo)
Our on disk path will require us to run a number of experiments with the storage format. We will move faster with a separate project.
Medium to long term we are very open to merge them.
> will require us to run a number of experiments with the storage format
I'd personally consider this a feature, if you want to maintain Postgres compatibility and allow your users to migrate in/out easily
Andrew is working hard on pgvector, and his goals are aligned to ours (and the industry's). We'll continue to support him as long as that's true.
> one of the slowest
Benchmarks are simply a snapshot in time - there's nothing fundamentally flawed with pgvector, it just needs more support and some better indexes (which are already being worked on)
[0] https://supabase.com/docs/guides/getting-started/architectur...
> Benchmarks are simply a snapshot in time - [...] it just needs [...] some better indexes
But the choice for IVF-flat indexes was made by the authors of pgvector, and that is one of the reasons why pgvector is slow: O(√n) optimal performance is just not great with large datasets, and this is especially true for the linked list approach chosen by the authors, as it results in large amounts of random IO.
You can't only implement a slow index and then shove the blame of the extension being slow on the index that was chosen by the extension authors: it is an inherent design issue. Sure, those can be fixed, but that doesn't mean it doesn't exist.
it would definitely be a design issue if the extension could only support a single index type, but Andrew already has plans to implement HNSW in 0.5.0:
0.5.0 according to the GitHub issue tracker has been more than a year in the making and it is not any closer to release.
If you say "no, pgvector is not slow, it is the index that is slow", while pgvector has been designed with only a single index type, then it is a design issue of pgvector in that the wrong type of index was chosen as the included index type.
That there are plans to solve this issue doesn't mean it is not an issue; indeed it validates my point that pgvector currently has design flaws.
Telling (potential) users to wait for 6 months for performance doesn't mean much if there are more performant alternatives readily available right now; a starving person isn't helped by a meal made available half a world away.
Scary that database engineers do not understand how reckless that is.
I am working quite a bit with normal and chained LLMs but so far haven't explored the retrieval route.
In my opinion, until LLMs learn to say I don't know, this is the way to do it in accuracy-first tasks.
Some tips:
- Most vector search is basically kNN under the hood, with some kind of compression. If you have too many embeddings in your DB, this starts to pull up irrelevant text very quickly. The key is to segment the DB using other data before doing the embedding search. Postgres extensions are good for this.
- The quality of the data you put into your embedding DB matters a lot.
- How you chunk text matters. Chunking by paragraph is much better than naive chunking, for example.
- This is a good benchmark for embedding models [2]
By segmenting data, is this as simple as adding a further condition to the SQL? E.g. if your embeddings belonged to a "client" you would start with WHERE client_id - x?
> How you chunk text matters. Chunking by paragraph is much better than naive chunking
Would love some more info on this. SO if you had a 400 word slab of text, it would be better to create m3 or 4 embeddings from that?
1. User types in the planned work (e.g., "replace discharge line on water pump")
2. The app embeds that and look for similar historical work in a vector database
3. We retrieve the most similar historical work and the safety observations that were made on those jobs
4. We summarize the major hazards or mistakes from those previous jobs using gpt-3.5-turbo and deliver that to the user