HNHacker News
TopNewBestAskShowJobs

duckworthd

452 karma · joined December 11, 2014

submissionscomments
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
Pictures are pictures, even as video frames :)
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
The model is trained to reconstruct what is observed, but not what is obscured. If you look closely at our videos, you'll notice some parts of the scene are blurry -- those parts weren't seen often enough to learn well. If you look at parts of the scene not observed at all, I'm not sure what you'd find.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
I'm actually quite new to the field, and I'm not even sure what to compare against nor how to compare it. What's typically measured and how?
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
The model does not explicitly learn to segment images. The answer is unfortunately more difficult to explain than a HN comment bears. I encourage you to read the paper for more details.

https://arxiv.org/abs/2008.02268

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
Consider? Yes. Try? Nope!
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
1) My bread and butter for the last 10 years has been machine learning. When all you have is a hammer...

2) We don't extract surface contours, we learn a volumetric radiance field! To oversimplify, we learn a (smooth) function that, given a position in space, produces the differential opacity and color at that space. To render an image from a camera viewpoint, we approximately integrate along rays emitted from each pixel of the camera.

Check out NeRF and our paper to learn more about this representation!

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
I don't know! I'm not familiar enough with the field to say.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
There is currently no way i'm aware of to accurately reconstruct a moving 3D scene. Sorry! Ask us again in a few years :)
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
I'm a bit new to the field myself, so I'm afraid I can't provide any contacts. Ask me again in a couple of years.

The method presented here wouldn't do well with your problem either. 3D reconstruction of moving objects is an unsolved problem!

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
I'm not sure I understand your question.

If the question is, "can you reconstruct a (static) scene from the frames in a video?", the answer is yes!

If the question is, "can you reconstruct a scene with people and other moving objects, and model them moving around too?", the answer is not yet.

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
> This looks amazing! Congratulations.

Thank you!

> What was the most challenging aspect of this?

Wow, that's hard to say! Our work truly stands on the shoulders of giants (Mildenhall et al, 2020). I can list off a few challenges: figuring out if an idea "kinda works" or "definitely works" or has a bug, figuring out how to measure progress, coordinating a group of 6 researchers living 9 hours apart, and assembling everything together for a simultaneous paper-website-video release!

> I'm curious to see how you performed edge detection on the transient objects and were able to isolate them so cleanly.

We don't! All of this comes by the magic of machine learning :). We train our model to attentuate the importance of "difficult" pixels that aren't 3D-consistent in the training set. We also partition the scene into "static" and "transient" volumetric radiance fields without explicit supervision. We do so by regularizing the latter to be empty unless necessary, and providing it with access to a learned, image-specific latent embedding. We discard the transient radiance field when rendering these videos, thus removing tourists and other moving objects.

> For some reason, the paper isn't loading, so feel free to say it's explained in detail there.

Well that won't do. Download it here: https://arxiv.org/abs/2008.02268. It's 40 MB, so your download bar may indicate it's almost done, but it actually has a good bit left to go.

> Do you plan on releasing code?

I hope so! As with most code, what ran on our machines may not run on yours. Migrating the code to open source will be a big effort. I hope what we describe in the paper is sufficient to build something like you see here.

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
At this point, this method is only good at reconstructing parts of the scene that are well-photographed. You'll notice that our video for Sacre Coeur has some blurry bits, particularly the staircase in front of the Basilica. That's because we learn to reconstruct what was seen, but aren't yet able to imagine what wasn't!

Does this work in reconstructing indoor spaces? Give it a shot and find out!

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
It's a little hard to describe from scratch, but let me do my best.

The method is unattended, in the sense that it's photos + camera parameters in and scene representation out. The photos should all be of the same scene (e.g. the Trevi Fountain). Once you have a scene representation, you can ask what the scene would look like from new camera angles with your choice of lighting.

Choosing camera angles is straightforward. You tell me where and what direction the camera is facing. The question then becomes, how do you specify your choice of lighting? The answer is, you can't do so directly. Instead, you provide a picture with the lighting you want, and with a little magic, we can find a way to imitate that lighting. The way we do is by finding a corresponding "appearance embedding" via numerical optimization.

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
That's a really cool idea! This technology does a fantastic job at reconstructing static scenes. The moving objects -- people, cars, even flora -- are out of scope here. Why? It's really hard to build a 3D model of something you only see from one direction.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
According to Wikipedia, "Photogrammetry is the science and technology of obtaining reliable information about physical objects and the environment through the process of recording, measuring and interpreting photographic images and patterns of electromagnetic radiant imagery and other phenomena."

I'd say that this research is in the field of photogrammetry.

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
The magic of this method is that we don't construct a "geometry" the same way one might think. There are no triangles or textures here. Instead, we train a machine learning model to predict the derivative of the color and opacity at every point in 3D space. We then integrate along rays emitted from the camera to render an image. It's similar to what's used in CT scans!
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
On the order of hundreds to low-digit thousands worked well for us. These photos contain a lot of occluders like tourists, and we needed to have enough views of the subject in question to build a good 3D scene representation.
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
Thank you :)
duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
We use images from the Image Matching Challenge 2020 dataset. If you look at the Appendix, we list how many images we use and the process by which they were chosen.

Download and have a look! https://vision.uvic.ca/image-matching-challenge/data/

duckworthd··on NeRF in the Wild: reconstructing 3D scenes from internet photography
Original author here. AMA!
duckworthd··on What I Learned from Working 32 Hours a Week
My $0.02.

I've been working a 4 day/week schedule for 1.5 years now. Initially, I took the time off to lead a nonprofit volunteering project but quickly got burned out (4 days work, 1.5~2 days nonprofit work = 5.5~6 days/week). I've since reallocated the extra day to hobbies and am quite happy with the setup. As I've explicitly taken an independent contributor role at work, I feel no need to be 100% available, and my colleagues respect that.

Overall, I'm very happy with my current schedule and will fight tooth and nail to keep it. If you can afford it and have the drive to do something with your extra free time, I highly recommend it.

duckworthd··on Neural Networks as Ordinary Differential Equations
A couple of questions,

1) In the motivation for the adjoint method, is u given along with A and C? I don't see how to compute u^T B without u.

2) How do a(t), da(t)/dt, and dG(..)/dh relate to A, B, C, u, and v from the motivation?

duckworthd··on JAX: Numpy with Gradients, GPUs and TPUs
Yes! Though this answer has some subtlety. tf.while() will run several iterations in parallel but this is not the same as _batching_ those same iterations. For that, you'll need to use the experimental parallel_for feature [0]. Using this should get you into roughly the same speed as JAX.

Tricky!

[0]: https://github.com/tensorflow/tensorflow/tree/b3e00739468080...

duckworthd··on JAX: Numpy with Gradients, GPUs and TPUs
There are two reasons,

1) Take a closer look at tf.hessian()'s implementation. Notice the tf.while() loop, calculating tf.gradient() for each coordinate of the first gradient? That doesn't happen in parallel! Plus, you need to know about TensorArrays!

2) Per-example gradients means an outside tf.while() loop over each example. Another linear slow down!

The difficulty I refer to is in wrangling conditionals in TF, and the trickiness of obtaining partial derivatives (rather than total derivatives) which JAX makes trivial.

There may be a

duckworthd··on JAX: Numpy with Gradients, GPUs and TPUs
My $0.02.

I've been using JAX for a while now. A paper I'm an author on (https://arxiv.org/abs/1806.09597, a follow-up to https://news.ycombinator.com/item?id=18633215) resulted in an algorithm that required taking second-derivatives on a per-example basis. This is extremely difficult in TensorFlow, but with JAX it was a 2-liner. Even better, it's _super_ fast, thanks to XLA's compile-to-GPU and JAX's auto-batching mechanics.

I highly recommend JAX to power users. It's nowhere near as feature-complete from a neural network sense as, say, PyTorch, but it is very good at what it does, and its core developers are second to none in responsiveness.

duckworthd··on Ask HN: Where to start on creating AI for games?
The best start I ever had to Artificial Intelligence was via UC Berkeley's CS188 course[1]. This course takes you through the very foundations of AI, from search to planning to basic probability theory. It's been imitated at universities around the world with its open lectures, slides, homework assignments and project code.

The projects are the very best part of this course. You start with a blank slate PacMan agent trying to pick up dots to something able to track & dodge ghosts without even seeing them (!). It's what inspired me to continue AI research 7 years down the road!

[1]: http://inst.eecs.berkeley.edu/~cs188/fa11/lectures.html

← PreviousPage 3 of 3