The concept and term "cultural marxism" has been grabbed by such a huge swath of people, most of whom have absolutely no idea it is or could be anti-Semitic (because you have to be pretty out there to associate Marxism with Jews to begin with), that claiming all or most uses of it are anti-Semitic is wrong.

There's just no way to paint that huge of a group with that brush unless you think it's so double-secret subconscious that the majority of the people using it don't even know they're anti-Semites.