An algorithm that recreates 3D objects from tiny 2D images
techcrunch.com
techcrunch.com
Well, surface rendering is how almost all 3d work is done already, that's definitely not a breakthrough. You can probably spend an entire career never dealing with voxels.
What this article never mentions (surprisingly) is that this paper is about neural networks. I'm not an expert, but as I understand the article, voxel representations have been the standard specifically when building neural networks to turn 2d images into 3d. The main idea is that you can build a network that only renders high-resolution voxels when close to the surface of the model, and renders very low-resolution voxels everywhere else (say, on the inside of the model). This means you can both represent much larger models memory-wise, but also that you're not having to run the NN computations on voxels that are not likely to change, since everything "inside" of the model you probably guessed correctly pretty quickly.
Here's the paper's abstract, much better at explaining itself than the article:
"Recently, Convolutional Neural Networks have shown promising results for 3D geometry prediction. They can make predictions from very little input data such as for example a single color image, depth map or a partial 3D volume. A major limitation of such approaches is that they only predict a coarse resolution voxel grid, which does not capture the surface of the objects well. We propose a general framework, called hierarchical surface prediction (HSP), which facilitates prediction of high resolution voxel grids. The main insight is that it is sufficient to predict high resolution voxels around the predicted surfaces. The exterior and interior of the objects can be represented with coarse resolution voxels. This allows us to predict significantly higher resolution voxel grids around the surface, from which triangle meshes can be extracted. Our approach is general and not dependent on a specific input type. In our experiments we show results for geometry prediction from color images, depth images and shape completion from partial voxel grids. Our analysis shows that the network is able to predict the surface more accurately than a low resolution prediction."
> So first his system renders a 3D reconstruction of the 2D image in very low resolution [...] Next, do a higher-resolution render of the area you kept.
Where apparently the author seems to think producing a 3D reconstruction of a 2D object is trivial even though that's what the paper is about.
Oh my god, I sound like my father.
For example the red pickup truck. There is no way the algorithm could create that model from that image only (depth of the trunk).
So my guess is that they use the tiny picture to search a database for similar pictures and then create a model with all that data.
The fact that all the objects are symmetric (at least in the examples) helps a lot.
If this algorithm manages to be as good as humans in estimating volumes of everyday objects, I can see it being helpful in Self Driving Cars.
I'm good at inferring the shape of something from a side view well enough that I can use it in some kludgy mental models, but down the page they're essentially asking the computer to look at something from the side and render a precise(ish) 3-D mechanical model of the object from memory. For example, I have what I think is a pretty good idea of what a Boeing 737 looks like, but if you asked me to draw it on a piece of paper it would look like a kinder gardener did it. What I'm good at is boiling down features and distinguishing a 737 from an Oscar Meyer wiener truck. Drawing a scale-accurate picture of it is a job for artists and savants.
See section 4 of the paper - they train on a set of 3d models from some 'ShapeNetCore' dataset, from which they generate sample inputs (renderings of the model with randomized viewpoint and lighting) and corresponding target outputs (voxelized model).
They train specialized networks for different classes of objects 'aeroplanes, chairs and car', so reconstructing on all classes at the same time probably still has some issues.
An interesting point about their use of this coarse-to-fine progression that the article omits: they do the same trick for training their net - first train to predict the coarse voxels, and when those work start predicting the next level.
It's not making a surface mesh, it's putting voxels on the surface. Again, it doesn't make that distinction very clear up front so I guess it's not targeted at people familiar with traditional 3D surface meshing.
This: "Computer, here's a 2D projection. Infer the likely 3D data that would produce the projection."
Look at the chairs' legs, where it's transforming the flat base of the rotating chair into something that looks as wheels, and where it's completely missing the bars connecting the legs in the second tall chair.
SPR is an algorithm already used to reconstruct 3d objects from a set of 2d images produced by an electron microscope, typically of a protein on a flat surface.
I haven't read the paper here, but I suspect a key part of the algorithm is that it can only reconstruct objects with symmetry planes: The airplane, chair, car are all symmetric across an axis. This greatly constrains the possibilities the algorithm has to search through. In em-reconstruction the user often specifies what they think the symmetries are beforehand.