Which do use attention? Any recommendations?
A lot of people wind up using models based purely on one or two benchmarks and wind up viewing embedding based projects as a failure.
If you do answer some of those I’d be happy to give my anecdotal feedback :)
I think this one is currently the top of the MTEB leaderboard, but large dimension vectors and a multi billion parameter model: https://huggingface.co/nvidia/NV-Embed-v1