Got it. So let's suppose we have an external renderer, then we could learn parameters to tweak rendered scene and then get rendered pixels, so we can calculate pixelwise errors from it and some target image we're trying to optimize for. In this way, do we still need differentiable renderer in your opinion?

Update:

It would require more training cycles and would not be as "atomic" as iterative tweaks but seems possible.

At same time, I wonder about making loss function talk with some external renderer would make it possible to mix both approaches.