Hacker News Reader

Comment by tommoor | Hacker News Reader

It feels like there are an influx of "vector databases" right now, I haven't had a strong answer out of anyone on why you'd be better off using these over Redis which offers vector storage with similarity search in a battle-tested OSS solution.

View on HN

I'm currently evaluating different vector stores and passed on Redis today after spending about a half day looking into it. Here's my reasoning

    1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs.

    2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented and cost me ~20 minutes. This is a pretty big problem given item 1.

    3. The `@redis/search` TypeScript types didn't allow some valid FS.SEARCH calls.

One thing to note is that for some use cases (IMO probably a large fraction of use cases right now), the vectors are derived data, so you don't necessarily need a super robust/battle-tested "database", you just need something with a simple happy-path API.

What do you think about Weaviate or Qdrant? There is a nice benchmarks overview with all the major players https://qdrant.tech/benchmarks/

I haven't really looked into either. So far I've tested pinecone, Redis, chroma, elasticsearch, and pgvector. I'm not really considering performance at all, just looking for something dead-simple to deploy and use. At the moment it looks like pgvector on supabase is the winner.

Yeah, if you don't need performance, then just take a simple ANN library. No need for a database at all. In terms of databases, Qdrant and Pinecone are the simplest I've tried so far. But Pinecone isn't open-source, not an option for on-prem. PGvector is too much imho, do not want to have all the other Sequel stuff if I just need an NN search.

Qdrant v1.1 was released recently and its quantization feature is just fantastic . See: https://github.com/qdrant/qdrant

Not trying to say you're wrong in your decision, but in the context of the post you're replying to, aren't these points arguments against the node client rather than redis itself?

1. node-redis or ioredis? If you are using a "modern IDE" (VS Code/WebStorm/any IDE with support for TS language server) you should get autocomplete for all the commands (in both of them).

2. The `@redis/search` package extends `@redis/client` with support for all RediSearch commands (you should get autocomplete for all RediSearch commands as well). If you want to get the "whole in one" package (support for redis vanilla + all redis modules) you can use the `redis` package instead.

3. Can you please share the command you are trying to run?

BTW, these examples might help:

1. https://github.com/redis/node-redis/blob/master/examples/sea...

2. https://github.com/redis/node-redis/blob/master/examples/sea...

3. Checkout `redis-om` https://github.com/redis/redis-om-node/tree/main

One positive thing I can say about Weaviate is that it's incredibly simple to get started with it, as it handles talking to the vectorizer(s) for you, and they provide many out of the box container images for them. With that we were able to integrate a similarity search into our product very quickly.

From what I can tell, its competitors generally don't do this, and you have to handle generating the embeddings and all the back and forth with the embeddings yourself, setting the barrier for integration much higher.

If you are at a level where you want to get a lot more control about embeddings, I agree that there it will often be better to just use the vector search capabilities that your existing solution like Redis/Postgres are getting.

Correct - this is actually a core focus of Weaviate. That's why it's not limited to only ANN but also has an inverted index, optional modules, etc.

Or Postgres with pgvector

https://github.com/pgvector/pgvector

Redis uses too much memory and supports only two NN algorithms FLAT (very slow) and HNSW if you start indexing millions of large vectors you will quickly understand the problem.

That being said, many of these DBs are overcomplicated for most use cases. Redis HNSW will work for many use cases.

Redis does fairly well single-threaded - iirc it does better than Milvus. The problem is, once you move to larger applications and want higher performance, its architectural limitations make it impossible to scale.

it uses a lot of memory because it's an in memory db, that's by design.

Enterprise has tiered memory. We have customers in the 50-100m range. benchmarked more than that as well. Single query latency will usually be better than most of the field. Also, were working on other variants of our index that will scale to 500m +

More to come. https://github.com/redisventures to keep up with our team

We've seen a huge uptick in users of Redis VSS. This thread is great feedback for us so thank you.

Vector search is still a semi active area of research. You can't get "battle tested" and optimal performance at the same time.

Reply on news.ycombinator.com