Have you experimented with late interaction?

https://jina.ai/news/what-is-colbert-and-late-interaction-an...

http://musingsaboutlibrarianship.blogspot.com/2024/06/can-se...

https://colbert.aiserv.cloud/