As with diffusion models in general, the point isn't the specific example but that it's generalisable.
5 million frames of video data with corresponding accelerometer data, and you get this for genuine photorealism.
5 million frames of video data with corresponding accelerometer data, and you get this for genuine photorealism.
And I thought I was fairly explicit about video data, but just in case that's ambiguous: the stuff you record with your phone camera set to video mode, synchronised with the accelerometer data instead of player keyboard inputs.
As for output, with the model as it currently stands, I'd expect a 24h training video at 60fps to be "photorealisic and with similar weird hallucinations". Which is still interesting, even without combining this with a control net like Stable Diffusion can do.