I also wished that was true while working on a fast web-based remote desktop client. Imagine if instead of a video stream via VNC, the UI was HTML or SVG diffs. That would be really fast! RDP does something similar but on a much lower-level.
I've been told it moved away from that as networks got faster, interfaces became more GPU-accelerated, and video encoding became hardware-accelerated, so that now it's not much different from VNC. But I don't know when or to what extent, or if RDP still uses any draw instructions at all.
I suspect that it transferred OS calls.
But even then, that would pale in comparison to SVG diffs from an SVG-native OS.