I guess you are not logging into the VM using a remote desktop client? But using the the spice client?

One of my favourite alternatives is using Steam Remote Play, you get the low latency, works for games at the cost of much higher bandwidth. But for a home environment this is fine.