452 karma · joined December 11, 2014
2) We don't extract surface contours, we learn a volumetric radiance field! To oversimplify, we learn a (smooth) function that, given a position in space, produces the differential opacity and color at that space. To render an image from a camera viewpoint, we approximately integrate along rays emitted from each pixel of the camera.
Check out NeRF and our paper to learn more about this representation!
The method presented here wouldn't do well with your problem either. 3D reconstruction of moving objects is an unsolved problem!
If the question is, "can you reconstruct a (static) scene from the frames in a video?", the answer is yes!
If the question is, "can you reconstruct a scene with people and other moving objects, and model them moving around too?", the answer is not yet.
Thank you!
> What was the most challenging aspect of this?
Wow, that's hard to say! Our work truly stands on the shoulders of giants (Mildenhall et al, 2020). I can list off a few challenges: figuring out if an idea "kinda works" or "definitely works" or has a bug, figuring out how to measure progress, coordinating a group of 6 researchers living 9 hours apart, and assembling everything together for a simultaneous paper-website-video release!
> I'm curious to see how you performed edge detection on the transient objects and were able to isolate them so cleanly.
We don't! All of this comes by the magic of machine learning :). We train our model to attentuate the importance of "difficult" pixels that aren't 3D-consistent in the training set. We also partition the scene into "static" and "transient" volumetric radiance fields without explicit supervision. We do so by regularizing the latter to be empty unless necessary, and providing it with access to a learned, image-specific latent embedding. We discard the transient radiance field when rendering these videos, thus removing tourists and other moving objects.
> For some reason, the paper isn't loading, so feel free to say it's explained in detail there.
Well that won't do. Download it here: https://arxiv.org/abs/2008.02268. It's 40 MB, so your download bar may indicate it's almost done, but it actually has a good bit left to go.
> Do you plan on releasing code?
I hope so! As with most code, what ran on our machines may not run on yours. Migrating the code to open source will be a big effort. I hope what we describe in the paper is sufficient to build something like you see here.
Does this work in reconstructing indoor spaces? Give it a shot and find out!
The method is unattended, in the sense that it's photos + camera parameters in and scene representation out. The photos should all be of the same scene (e.g. the Trevi Fountain). Once you have a scene representation, you can ask what the scene would look like from new camera angles with your choice of lighting.
Choosing camera angles is straightforward. You tell me where and what direction the camera is facing. The question then becomes, how do you specify your choice of lighting? The answer is, you can't do so directly. Instead, you provide a picture with the lighting you want, and with a little magic, we can find a way to imitate that lighting. The way we do is by finding a corresponding "appearance embedding" via numerical optimization.
I'd say that this research is in the field of photogrammetry.
Download and have a look! https://vision.uvic.ca/image-matching-challenge/data/
I've been working a 4 day/week schedule for 1.5 years now. Initially, I took the time off to lead a nonprofit volunteering project but quickly got burned out (4 days work, 1.5~2 days nonprofit work = 5.5~6 days/week). I've since reallocated the extra day to hobbies and am quite happy with the setup. As I've explicitly taken an independent contributor role at work, I feel no need to be 100% available, and my colleagues respect that.
Overall, I'm very happy with my current schedule and will fight tooth and nail to keep it. If you can afford it and have the drive to do something with your extra free time, I highly recommend it.
1) In the motivation for the adjoint method, is u given along with A and C? I don't see how to compute u^T B without u.
2) How do a(t), da(t)/dt, and dG(..)/dh relate to A, B, C, u, and v from the motivation?
Tricky!
[0]: https://github.com/tensorflow/tensorflow/tree/b3e00739468080...
1) Take a closer look at tf.hessian()'s implementation. Notice the tf.while() loop, calculating tf.gradient() for each coordinate of the first gradient? That doesn't happen in parallel! Plus, you need to know about TensorArrays!
2) Per-example gradients means an outside tf.while() loop over each example. Another linear slow down!
The difficulty I refer to is in wrangling conditionals in TF, and the trickiness of obtaining partial derivatives (rather than total derivatives) which JAX makes trivial.
There may be a
I've been using JAX for a while now. A paper I'm an author on (https://arxiv.org/abs/1806.09597, a follow-up to https://news.ycombinator.com/item?id=18633215) resulted in an algorithm that required taking second-derivatives on a per-example basis. This is extremely difficult in TensorFlow, but with JAX it was a 2-liner. Even better, it's _super_ fast, thanks to XLA's compile-to-GPU and JAX's auto-batching mechanics.
I highly recommend JAX to power users. It's nowhere near as feature-complete from a neural network sense as, say, PyTorch, but it is very good at what it does, and its core developers are second to none in responsiveness.
The projects are the very best part of this course. You start with a blank slate PacMan agent trying to pick up dots to something able to track & dodge ghosts without even seeing them (!). It's what inspired me to continue AI research 7 years down the road!
[1]: http://inst.eecs.berkeley.edu/~cs188/fa11/lectures.html