60hz is great while looking at a screen but may not be enough for keeping a good illusion going while you move around space. hololens has a similar setup that renders triangles at whatever rate it can but updates the position on the display at 240hz (that is, even if the next frame of an animation is lagging, its position in space will be adjusted as you move)

imo the illusion was rock solid, extremely challenging given that the display was transparent so you had to keep up with the real backdrop moving - vision pro and all passthrough devices get to fake it but at the cost of proprioception as you said