I haven't noticed this behavior in such extreme ways as you. I've worked with many different embeddings using this library and the OpenAI API for data normalization, extraction and structuring tasks.
On one hand it's super impressive what you can get out of these embeddings. On the other I'm with you that there seems to be a missing piece to make them work great.
An additional MLP or Siamese Network works really well but it seems like there should be something easier and unsupervised given your set of vectorized samples.
Thinking out loud, could PCA help with this by centering the data and removing dimensions that mostly model noise?