HNHacker News
TopNewBestAskShowJobs

duckworthd

452 karma · joined December 11, 2014

submissionscomments
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Thank you!
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Check out our explainer video for answers to this question and more! https://www.youtube.com/watch?v=zhO8iUBpnCc
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
It would be my dream to make capturing 3D memories as easy and natural as taking a 2D photos with your smartphone today. Someday!
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
> This is __really__ stunning work

Thank you :)

> Is it a consequence of input photo quality, or something else?

It's more a consequence of spatial resolution: the bigger the space, the more voxels you need to maintain a fixed resolution (e.g. 1 mm^3). At some point, we have to give up spatial resolution to represent larger scenes.

A second limitation is the teacher model we're distilling. Zip-NeRF (https://jonbarron.info/zipnerf/) is good, but it's not _perfect_. SMERF reconstruction quality is upper-bounded by its Zip-NeRF teacher.

duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Not at the moment but an intrepid hacker could surely extend our JavaScript code and put something together.

UPDATE: The code for our web viewer is here: https://github.com/smerf-3d/smerf-3d.github.io/blob/main/vie...

duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Right-o. The web viewer is swapping assets in and out of memory as the user explores the scene. The Network and disc requirements are high but memory usage is low.
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Thank you :)
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
In principle, there's no reason you can't fit multiple City blocks at the same time with Instant NGP on a regular desktop. The challenge is in estimating the camera and lens parameters over such a large space. I expect such a reconstruction to be quite fuzzy given the low space resolution.
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
> Are there opportunities, where they exist, to not use optimization or tuning methods for reconstructing a model of a scene?

If you know a way, let me know! Every system I'm aware of involves optimization in one way or another, from COLMAP to 3D Gaussian Splatting to Instant NGP and more. Optimization is a powerful workhorse that gives us a far wider range of models than a direct solver ever could. > Can we still achieve the great look and details of RF and GS without paying for an expensive reconstruction per instance of the scene?

In the future I hope so. We don't have a convincing way to generate 3D scenes yet, but given the progress in 2D, I think it's only a matter of time.

> Are there ways of greedily reconstructing a scene with traditional CG methods into these new representations now that they are fast to render?

Not that I'm aware of! If there were, I think these works should be on the front page instead of SMERF.

duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
;)
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
I can't predict the future, but I imagine soon: all of the tools are there. The reason we didn't develop for VR is actually simpler than you'd think: we just don't have the developer time! At the end of the day, only a handful of people actively wrote code for this project.
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Glad you liked our work!

1) Around 100-150 if memory serves. This scene is part of the mip-NeRF 360 benchmark, which you can download from the corresponding project website: https://jonbarron.info/mipnerf360/

2) Between 12 and 48 hours, depending on the scene. We train on 8x V100s or 16x A100s.

3) The time for preparing assets is included in 2). I don't have a breakdown for you, but it's something like 50/50.

4) Nope! A keen hacker might be able to do this themselves by editing the JavaScript code. Open your browser's DevTools and have a look -- the code is all there!

duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Pardon our dust: "images" is a bad name for what's being loaded. Past versions of this approach (MERF) stored feature vectors in PNG images. We replace them with binary arrays. Unfortunately, all such arrays need to be loaded before the first frame can be rendered.

You do however point out one weakness of SMERF: large payload sizes. If we can figure out how to compress them by 10x, it'll be a very different experience!

duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
The pretrained models are already available online! Check out the "demo" section of the website. Your browser is fetching the model when you run the demo.
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Thank you :). I'm glad to hear it! Which model are you using?
duckworthd··on SMERF: Streamable Memory Efficient Radiance Fields
Gaussian Splatting is heavily inspired by work in radiance fields (or NeRF) models. They use much of the same technology!
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
We did not aim to speed this part of NeRF up. Check out Neural Sparse Voxel Fields (https://arxiv.org/abs/2007.11571) for some effort in that direction. It's 10x faster, but there's still another 10x to go till you get video frame rates :)
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
A good question! And a problem yet to be solved!
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
The work we build off of, NeRF, does. While there's nothing preventing NeRF-W from also representing reflections, we find it captures a more matte picture of the object.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
Yes, the images need to be posed. We use COLMAP to obtain camera pose.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
NeRF's (and all of photogrammetry's) bread and butter is 3D consistency -- that is, seeing the same object from multiple angles. A 360 degree photo from a fixed position just won't do. As to how to select the best camera angles...I'm not sure. I believe there is research in this area for classical photogrammetry techniques, but I'm not familiar enough to point you to a body of work.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
NeRF was first published in March! Give us time :)
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
We have depth! Check out our depth video: https://youtu.be/yPKIxoN2Vf0?t=146
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
Another related project by our friends in NYC: https://twitter.com/Jimantha/status/1289184432553734144
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
For that, you'll need to convert the representation we have (volumetric radiance field) to on your 3D printer can understand (a mesh?). The NeRF authors use the marching cubes algorithm to do just that. Check out their website: https://www.matthewtancik.com/nerf.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
Answered most your other questions below in another comment.

> and what are z(t) r(t) in equation 5,6?

r(t) is a position in 3D space along a camera ray of the form, r(t) = origin + t * direction.

z(t) is the output of our first MLP. Think of it as a 256-dimensional vector of uninterpretable numbers that represent the input position r(t) in a useful way.

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
Our code is not released yet. If your data is captured without occluders and in RAW format, you don't need the enhancements we propose :). NeRF can do amazing things with clean data!
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
It is provided by the dataset we use, but given a new dataset, you can use off-the-shelf tools to obtain it yourself! Check out COLMAP, it's super duper cool: https://colmap.github.io/.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
> so why does it use multi-layer perceptron? is it the same as ANN? why not calling it ANN? Does it have activation?

According to Wikipedia, "A multilayer perceptron (MLP) is a class of feedforward artificial neural network (ANN)." We're being more specific about what we use :)

> There is another thing I don't understand. Traditional volume data is a map of position to density (or color). There doesn't seem to be the need for a direction as an input. Why does the network need a direction? Why can't we get a density and a color given a position?

Volume data of this form is unable to express the idea of view-dependent reflections. I admit, we don't make much use of that here, but it does help! See NeRF for where it makes a big, big difference: https://www.matthewtancik.com/nerf

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
It's hard for me to say how precise camera position and direction needs to be. We use COLMAP to estimate both via multi-view stereo.
← PreviousPage 2 of 3Next →