Yes, same here. I feel I am very sensitive to this (among other sensory things, honestly).

In my case the Steam overlay reported either 13 or 25ms (can't remember) total latency (including video encoding/decoding) and the latency to the game server was 5ms. This put my total latency at around 30ms max which wasn't too bad and not too different from other players. Granted, it's not a perfect comparison (other players have high network latency while mine was input latency which can't be compensated by the game). It did however impressively exceed my expectations (I expected it to be completely unplayable) so I suggest you try it.

Maybe 5ms of network latency, but it takes a lot longer to encode your mouse movement into a packet and send it over a wire, add 5ms, then wait for the cloud data center to route it to the right VM, decode your packet, send it to the game, capture the framebuffer, and send it back to you. The end to end roundtrip is more than just network latency.

Anecdote: I've used a lot of cloud gaming providers with fiber to the home, and while the experience is great, it's very noticeably laggier in first person shooters compared to going from 90fps to 60fps. It's a different kind of latency... beyond 60+ FPS, there are more frames in between, and so motion is smoother. But even at 60fps, your "time to first frame (update)" is still very low, compared to over the internet (or even to a local Steam remote desktop). Cloud gaming at 100Hz/FPS still feels laggier than 60fps local because of that mouse lag.

Modern graphics cards are very good at rapidly encoding frames using hardware accelerators, but it's still nowhere near as good as drawing directly to the screen. Even the fastest multi-gig networks are orders of magnitude slower than your internal buses and ports, especially in latency.