driving the display is just a part of it - they need to do super accurate SLAM, object recognition/machine learning inference, and if the visuals are interactive and not pre-rendered then all the lighting, texturing, rigging, physics etc of the rendered models - all this on the same cpu/GPU.
It's just about doable on desktop VR setups with latest desktop GPUs