> But what’s even more impressive - extremely impressive - is that the system decided that the body would go up before going back down between these two poses! (Which is why it’s having trouble with the right arm in the first place! A feature matching system wouldn’t have this problem, because it wouldn’t realize that in the middle position, the body would go up, and the right arm would have to be somewhere. Struggling with things not visible in either input keyframe is a good problem to have - it’s evidence of knowing these things exist, which demonstrates quite the capabilities!).... This system clearly learned a lot about three-dimensional real-world movement behind the 2D images it’s asked to interpolate between.
I think that's an awfully strong conclusion to draw from this paper - the authors certainly don't make that claim. The "null hypothesis" should be that most generative video models have a ton of yoga instruction videos shot very similarly to the example shown, and here the AI is simply repeating similar frames from similar videos. Since this most likely wouldn't generalize to yoga videos shot at a skew angle, it's hard to conclude that the system learned anything about 3D real-world movement. Maybe it did! But the study authors didn't come to that conclusion, and since their technique is actually model-independent, they wouldn't be in a good position to check for data contamination / overfitting / etc. The authors seem to think the value of their work is that generative video AI is by default past->future but you can do future->past without changing the underlying model, and use that to smooth out issues in interpolation. I just don't think there's any rational basis for generalizing this to understanding 3D space itself.
This isn't a criticism of the paper - the work seems clever but the paper is not very detailed and they haven't released the code yet. And my complaint is only a minor editorial comment on an otherwise excellent writeup. But I am wondering if the author might have been bedazzled by a few impressive results.