The camera is only used to track the controller, the actual 3D data is pulled from the game files. The Quest API does let you layer elements onto the passthrough view[0], so it could work, the game would just hover in front of you instead of tracking the controller.

[0] https://developer.oculus.com/documentation/unity/unity-passt...