Depending on bandwidth, you could probably stream an over-rendered viewport (e.g. 3D skybox) that allows for local-device-magnitude latency for critical functions (perspective change & location translation), while being backed by streaming servers (especially of the edge variety).