https://www.youtube.com/watch?v=C4IT1gnkaF0 D3GA: Drivable 3D Gaussian Avatars
https://www.youtube.com/watch?v=q5hHZrRUV-k Animatable Gaussians: Learning Pose-dependent Gaussian Maps for High-fidelity Human Avatar Modeling
I can't say I'm a fan of replacing well defined systems with a big blob of weights that nobody really understands.
If the NN doesn't get it right, it's absolutely not the end of the world. First of all, triaging the failures usually turns up something interesting / some connecting thread. And as for fixing issues, your options are typically:
1. Retrain with more examples to help it get it right (easier these days since you can take advantage of synthetic data / self- or semi-supervised approaches). Specifically it will help to generate or find examples that approximate the failures.
2. Shape your loss to penalize undesirable behavior / tendencies you've identified in triage
3. Add inputs or outputs that help your net solve the problem better (e.g. side inputs that may help the net not overfit to its available data)
These "inscrutable black boxes," while hard to edit directly, are not impossible to evaluate which means you can just measure to see if they're better than more explicit systems. And most real world systems are patchworks of ML + heuristics to give some desired level of performance.
Furthermore there is a big field of folks dedicated to elucidating what your net is learning (e.g. saliency). The tools are pretty cool -- for example in computer vision you can visualize whether your net is overfitting to background data in your training examples.
A valid critique of all of this is simply that it's expensive. Very expensive to build teams to warehouse and clean huge amounts of data, then build unbiased / fair model evaluations, then build models, then integrate them into real world stuff, usually to be used in conjunction with non-learned algos.
Another valid critique is that many companies have started using models they can't retrain or fix (e.g. off the shelf LLMs). This is dangerous territory since you lose so many of the options I outlined above.
I guess if you just don't care about the details of what you're making then AI could lower the barrier to producing something mediocre, as it has with 2D image generation, but that's damning with faint praise.
To me this sounds a lot like how people complained in 80s(?) about compilers generating worse code than expert assembly programmers. Yes, it is true, but also quantity can trump quality. And allowing machine to do bulk work, even if mediocre, allows humans to focus on key parts; in code some hot kernels, in 3d modeling maybe some hero assets. Also like compilers have evolved lot in the past decades, I don't see reason why these ML models/NNs/etc could not improve on a similar trajectory. Note that I'm not saying that they inevitably will improve to superhuman levels, but just that at this point they don't seem something that can be completely dismissed.
Furthermore I don't think it's faint praise if tool can enable non-experts to create compelling 3D experiences easily even if they are not perfect and photorealistic. Again I draw parallels to programming; lowering the skill level needed has allowed huge amount of valuable programs to be written even if the code behind them is complete crap
Without being an expert in this area, I imagine one such way would be to train a second neural net that learns to regress traditional workflow parameters to best approximate the NeRF output, then let a user post-edit those parameters to fine-tune.