I suspect that a lot of the people saying that sort of thing is impossible just haven't tried it. Most of my friend group uses Shadow for everything nowadays, and I can't see myself switching back to maintaining my own hardware.
FPS games like Skyrim work great, even latency-sensitive games which require quick reactions like Thumper do pretty well. I get lower latency / better image quality using Quest+VD than I do a standard 1080p stream on my Steam Link.....
I think that the VR video stream is already a 180 degree stereoscopic format helps with the latency; you're only ever looking at a subset of the stream so when you move your head around it's the same latency as local VR.
Only noticeable thing is if you move your head really quickly to over your shoulders it can take a few milliseconds for the footage to appear.