I also have a 120Hz Mac, and the animation is indeed slower in 120Hz mode. In my opinion, the animation crossed the line from "too slow but bearable" to "unbearable" with 120Hz. It is as you say; it's not really the animation itself that's the problem, but the delay from when I tell my machine "switch to this other workspace" until the focus switches to a window on that other workspace. The animation has this horrible ease-out effect where the last few centimeters take what feels like forever.

Getting a 120Hz Mac actually completely changed my whole macOS philosophy. I used to use spaces extremely heavily. I now almost don't use them at all, preferring window switching with cmd+tab instead.

The infuriating thing is that almost all discussion on this on the web just says "turn on reduced motion". Not only should that be unnecessary; it doesn't even fix the problem! Sure, there's no longer a sliding animation, but there's now a fade animation instead which takes just as long.

It's completely incomprehensible that Apple hasn't fixed this.

Sadly, solutions like BetterTouchTool and InstantSpaceSwitcher won't work for me because I prefer to use my trackpad to switch spaces.

EDIT: I actually recorded and compared the switching speeds a while ago: https://old.reddit.com/r/MacOS/comments/rfmg4e/workspace_swi.... Apologies for the choppy recording, QuickTime screen recording is not very good; but it gets the point across.