Obviously in a given model (i.e. word2vec) the representation of animal and cow will be correlated. I was wondering, how can I extract the orthogonal component of the word "cow" that is not driven by "animal" and capture its sentiment?
The mini-V2 (default sentence transformer ones) are far superior for these tasks than many others.
The query-document snippet style ones (trained on MS-Marco) are great for matching definitions etc or QA.
The implicative and other more semantic embedding types are ok but usually too slow to be practical, especially cross-encoders.
The Word2Vec ones are very fast, but not deep. The main usage of terms dominates in cases of polysemy which will obstruct your goals. So I recommend exploring cosine similarity of SBERT.
it should be said given the predominance of GPT in these tasks, besides for academic purposes, these techniques are not particularly useful these days - GPT will be much more useful for almost every end task (every task I've tried it has been, but I accept their could be exceptions, as somebody who used to compete well on Kaggle for sentence embedding/classification kind of tasks over the last few years)
You can subtract or add embeddings or pool embeddings are mentioned but these won't exactly remove concepts. What you would do better to do is remove the embedding which represents the pooled average of other animal nouns (Dog, Cat, etc) rather than "animal", because "animal" will be used as a verb, will be used as a collective, it won't probably get to what you're aiming for.