Interesting. Perhaps my idea of this being inserted in to existing algorithms would not work.

However I do ultimately seek a low accuracy “visually approximate” 3D scene that I could use for simulation purposes. I guess I could rephrase my desire as: I’d love to see this kind of approach used to train an end to end deep learning photogrammetry system. I feel like the parallel nature of neural nets as well as their ability to approximate results could result in a much less computationally intensive solution to my photogrammetry desires.

(I want to train my four wheel drive robot to follow forest trails using the training method described in the “world models” research paper, which requires a simulation to work.)