Hacker News Reader

Comment by rlt | Hacker News Reader

For truly immersive VR video it seems we’ll need to support moving your head in 6 degrees of freedom, so you can peak around objects etc. We’re a long ways off from it being realtime, but I can imagine generative AI being used to interpolate frames from an array of cameras spaced fairly widely apart.

View on HN

Lytro had this more-or-less figured out, before they ran out of money.

https://www.roadtovr.com/exclusive-lytro-reveals-immerge-2-0...

Google had a complete system for capturing volumetric light fields in 2020: https://augmentedperception.github.io/deepviewvideo/

There was a lot of work on big light-field capturing rigs in the past, but honestly, with the current advancements in neural radiance fields, gaussian splatting, meshing and other scene reconstruction techniques even monocular capture can generate pretty amazing results. I'm sure that having ultra-high res binocular video could be converted into some pretty high quality free-look output (that can at least be played back in real-time) on high-powered mobile devices could be possible today (w/ just some additional optimization/engineering work).

Recent research examples:

* https://vidu4d-dgs.github.io/

* https://fudan-zvg.github.io/4d-gaussian-splatting/

* https://guanjunwu.github.io/4dgs/

* https://aoliao12138.github.io/VideoRF/

There's a cool VR demo experience showcasing this on Steam: Welcome to Light Fields[0].

[0] https://store.steampowered.com/app/771310/Welcome_to_Light_F...

Yes, this is the closest to what I would consider a true VR photo. Stereoscopic images are gimmicky and feels utterly restrictive compared to this.

From the same lab 2 years later, there's "Immersive Light Field Video with a Layered Mesh Representation" which is the same, but instead of photo it's video.

Interesting! Thanks for the pointer.

4D Gaussian splatting will probably get us there, there are many research papers about it for example:

https://fudan-zvg.github.io/4d-gaussian-splatting/

But even with that, it would allow only limited movement because if you move to much in the reconstructed space then it would have to show you occluded areas that has been never recorded. But for that case generative AI could be used to fill in the the gaps, which means it wouldn't be true to life reconstruction anymore, but it could be still useful where that isn't that important.

In the Vision Pro demo appointment reel (i.e. anybody can demo the headset for free), everything is 6DOF, most is just stereo. Immersive VR is interactivity, which is more about good UX / gamification and less about using a dual-8k stereo capture device.

To some degree it's already being done on the headset in realtime to produce a passthrough image with proper depth

Not really, that image is stereo but it isn’t 360 degree and it can’t support arbitrary viewing angles and positions after the fact. It works perfectly for passthrough because the headset cameras are physically locked to your head.

It does support arbitrary viewing angles in the sense that IPD is accounted for. It generates a view for your specific eye position. It handles in painting around the user's hands as well.

Of course it's not magic so the views it can generate are limited.

Reply on news.ycombinator.com