My newer desktop (2020 era with a 3070) has 4x 4k monitors attached running XFCE and I have never noticed the lag you speak of. I don't run external monitors on it but my thinkpad x200 with a core 2 duo also does great with xfce.
I have no doubt the issues you speak of exist in theory but they do not seem to matter in practice.