Hacker News Reader

Comment by arecurrence | Hacker News Reader

I was using pinecone before installing pgvector in Postgres. Pinecone works and all but having the vectors in Postgres resulted in an explosion of use for us. Full relational queries with where clauses and order by etc AND vector embeddings is wicked.

View on HN

Why do you use pgvector instead of pgANN? My understanding is pgANN is built with FAISS. When I compared pgvector with FAISS, pgvector was 3-5x slower.

https://github.com/netrasys/pgANN

If I understand correctly, pgAnn is using a standard "cube" extension: https://github.com/netrasys/pgANN#setup

There is certainly a wide variety of problems today for which pgvector is unsuitable due to performance limitations... but fear not! This is an area that is getting significant focus right now.

A marqo.ai dev is currently working on adding HNSW-IVF and HNSW support https://news.ycombinator.com/item?id=35551684 and the maintainer has recently noted that they are actively working on an IVFPQ/ScaNN implementation https://github.com/pgvector/pgvector/issues/93

The pgAnn creator actually asked about performance a month ago here https://github.com/pgvector/pgvector/issues/58

Expect to see performance improve dramatically later this year.

This hits home, it is a big ask to keep data in sync for yet another store. We already balance MS SQL and Algolia and all the plumbing required to catch updates, deletes, etc. adding another feels like a bridge too far. Hopefully MS will get on this train at some point and catch up to postgres.

What does it do is it just storing vectors and cosine similarity? Why is there a whole new category of db for that

How is disk usage in pgvector?

There's a section in the readme describing vector space https://github.com/pgvector. . Also, the maintainer is actively working on product quantization https://github.com/pgvector/pgvector/issues/93

Supabase was also asking for sparse vectors https://github.com/pgvector/pgvector/issues/81

Speaking of the repo, they have a number of features they want to add if anyone is interested in contributing, there's lots of room for advancement. Many of these features already have active branches https://github.com/pgvector/pgvector/issues/27

Do you know of anything similar available for SQL Server 2019?

Reply on news.ycombinator.com