This is nothing new- Google's "Cardboard Camera" app has been around for a few years and does almost exactly this same thing. https://play.google.com/store/apps/details?id=com.google.vr....
Until you look down. Or until you have a different eye distance than average. Or you try to move a bit and there are things close enough to make you dizzy.
Cardboard didn't need any of those fancy things, of course. It was bad enough with abysmal latency.