What are you running for audio? pipewire+jack, pipewire, jack2, pulseaudio? I wonder if it's from latency. Pulseaudio is the most common but if you do any audio engineering or play guitar etc with your machine we all use jack protocol for less latency.

https://linuxmusicians.com/viewtopic.php?t=25556

Could be completely unrelated though, RDP sessions can definitely act up, get audio out of sync etc. I try to never do pass through rdp audio, it's not even enabled by default in the mstsc client IIRC but that may just be a "probably server" thing.