Support different models, OpenAI embedding model and open source models (currently SentenceTransformer). Full Lucene search ability + Vector search.
Anyone thinks this is a useful project?
Since Lucene already supports ANN vector search, I am curious how this project is different than what's in core.
Lucene has 2 limits:
1. Only supports 1 vector per doc. The doc content will have many vectors. You will have to store the chunks of one doc as different docs in Lucene. If you want to leverage Lucene's fields to store doc's metadata, such as title, tags, etc, and fetch them in search, need to work around. Also if you still want the traditional keyword search ability over doc content, better to keep as one doc.
2. Max vector dimension is 1024. OpenAI embedding is 1536. Lucene has discussion on increasing it, but there are concerns.