I'd love to hear John Carmacks' take on this. They had to solve very similar problems when developing this helmet:
extremely low-latency head & eye-tracking, managing depth-of-field, integrating multiple input streams, providing non-distracting information overlays etc.