I would suggest that anyone trying a real comparison of vector DB's consider the following
- necessary functions / use cases (eg prefiltering, dense search)
- embeddings version management
- anticipated embedding size (the article only considers glove-100 on ANN-benchmarks, which is quite different from openai-ada-002 1536 - both in terms of their output distribution and the vector size)
- required precision / recall
- required ingestion speed
- required ingestion throughput / time to ingest periodic updates
- required query speed (percentiles, not average!)
- required query throughput
- required RBAC, data privacy, active-active, etc
...and so much more. ANN-benchmarks is a good start for thinking about this but remember that actual throughput is quite different from whatever you see in the algorithms benchmarking!