We've had good success with Vespa [1] for this usecase, both using it to rank by vector similarity and apply business logic oriented filters.

Search latencies are fairly low, even with constantly updating documents.

[1] https://tech.okcupid.com/vespa-vs-elasticsearch-for-matching...