Hacker News Reader

If you tried to repro these results (including time & space constraints) using traditional photogrammetry, you would be sorely disappointed.

Photogrammetry is great if you have a very solid object that is not shiny or translucent at all. You get a lot of surface color micro-detail and a bit of bumpy meso-detail.

But, if something is fuzzy, hairy, or lacey or smokey you are straight-up out of luck. Don't even try.

If it is shiny, it can be difficult to capture at all --let alone capture the shine. Material capture techniques that are not "chalk sculpture" are rare, very limited and usually experimental.

NeRFs however are pretty much a photograph that you can walk around in. They have about as much structure as a photograph ;) But, that lets them not care about the mathematical definition of your scuffed-up, lacquered, iridescent, carbon fiber mirror frame and just show it as it looks from whatever angle.

Maybe NeRFs can be used as an intermediate step to reconstruct the scene, and then extract the surfaces and their materials to more conventional representations like meshes, textures, refraction index, etc. I guess the main benefit is that it fills in the undersampled areas in a scene, whether that's an occlusion, a reflection angle, or something else.

My main problem with them is that it seems as if all the data is unstructured and interdependent, not like pixels, voxels, or similar where you can clearly extract and manipulate parts of the data and know what it means. To use your photograph example, a digital photo is a simple grid of colored points, and it's easy to change them individually. A regular 3D scene is a collection of well defined vertices, triangles, materials etc, that is then rendered into a digital photo using a easy to describe process. A NeRF on the other hand seems to be more like enter camera pose => magic => inferred image.

Maybe I'm overthinking it and it doesn't have to be as general as out current formats, maybe a binary blob that can represent a static scene is fine for plenty of applications. But it feels needlessly complicated.

https://www.matthewtancik.com/nerf is able to sample the 3D volume somehow. "We can also convert the NeRF to a mesh using marching cubes."

That's more interesting than I realized. In this example, I assumed that the model was generating some sort of 3D mesh representing the woman. Is that not at all the case? Would this technique be unable to generate a model or volumetric information despite being able to reasonably render her from many directions?

No, there is no mesh. A NeRF is a neural network trained to work as a function f(x, y, z, θ, φ). You put in your viewing position (x, y, z) in 3D space and the direction (θ, φ) you're looking into (where θ and φ are the angles for up/down and left/right, respectively), and the function will output a tuple (r, g, b, σ) of the colour (r, g, b) and the material density (σ) of whatever you see at the pixel in that direction and from that position.

You can generate a mesh from the density information this function gives you, but for that you need to discretise the continuous densities you get out.

Minor correction: it's not your (i.e. the camera's) XYZ position what you input, but the position of the point whose RGBA you're trying to render.

Also, the method is lossy as hell. Not something you'd ever use for an intermediate representation.

The pros in the vfx industry still all use reconstructed geometry. And yes, animating or cropping a Nerf is painful.

In my opinion, Nerf is more about showing progress in making AI memorize 3D scenes and the hope is that this will lead to actual understanding sometime in the future.

What happens if you want to animate part of a NeRF, or crop it, or change it in any way? Do you have to throw away all trained coefficients and start again from training data?

You don’t change NeRF (the model). You change the point of view of an observer.

I mean, this is the parent posts point; the use case for a static photo or a static 3d nerf is pretty limited.

With other structured data compositing and animating is relatively trivial.

It turns out that people have approached this problem before and you can composite nerf too (1) by sampling different functions over the volume.

…but, let’s not pretend.

The complaint is entirely valid. You’re taking a high resolution voxel grid and encoding it into a model.

Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf.

Practically speaking, the applications you can use this for are therefore reasonably limited right now.

[1] - https://www.unite.ai/st-nerf-compositing-and-editing-for-vid...

> Working with simple voxel data let’s you do all kinds of normal image manipulation techniques, and it’s not clear how you would do some of those with a nerf.

Any image transformation you can do on voxels you can straightforwardly transfer to nerfs. Voxel data is just a lookup table from discrete positions to material properties like color and density. When you apply a transform, you change the inputs (e.g. multiplying them with a rotation matrix) or the outputs (e.g. changing the color). If you want to do the same thing with a nerf that maps continuous positions and directions to material properties like color and density, just transform the inputs or the outputs.

The major difference is that with voxel data you can easily do output-modifying transformations directly on the stored representation, while for nerfs it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model.

> Any image transformation you can do on voxels you can straightforwardly transfer to nerfs

No.

> it might be cheaper to do it on the fly instead of redoing the training procedure to bake the change into the model.

I think it’s a bit more complex than you imagine; it’s not “cheaper/not cheaper”; it’s literally the only way of doing it.

If you have a transformation f(x) that takes a pixel array as in input and returns a pixel array as an output, that is a trivial transformation.

If you have a transformation that takes a vector input f(x) and returns a pixel output, it’s seriously bloody hard to convert it to a “good” vector again.

Consider taking a layered svg and applying a box blur.

Now you want an svg again.

It’s not a trivial problem. Lines blur and merge, you have reconstruct an entirely new svg.

Now you add the constraint in 3d; you can never have a full voxel representation in memory even temporarily because of memory constraints.

At best you’re looking at applying voxel level transformations on the fly to render specific views, and then retrain those into a new nerf model.

I think that counts as … not straightforward.

Doing all your transformations on the fly is a lovely idea, but you gotta understand reason nerf exists is that the raw voxel data is too big to store in memory. It’s simply not possible you can dynamically run a image processing pipeline over that volume data in real-time. You have to bake it into nerfs to use it at all.

> Consider taking a layered svg and applying a box blur.

> Now you want an svg again.

> It’s not a trivial problem. Lines blur and merge, you have reconstruct an entirely new svg.

Nerfs are not svgs.

Consider taking a nerf and applying a box blur. Easy, a box blur on voxel data takes multiple samples within a box and averages them together, so to do the same thing to a nerf just take multiple samples and average them together.

That does get slower the more samples you need, but you never have to materialize a full voxel representation.

You still have to process every voxel.

> Doing all your transformations on the fly is a lovely idea, but you gotta understand reason nerf exists is that the raw voxel data is too big to store in memory.

> It’s simply not possible you can dynamically run a image processing pipeline over that volume data in real-time.

You are right, this is why approaches like plenoxels are vastly faster than nerf. They combine the optimization approach of neural nets, but combine it with a simple and regular data representation of the scene.

https://arxiv.org/pdf/2112.05131.pdf