I don’t understand why these world models aren’t designed to be backed by an updatable world state. It could probably just lay out the world onetime and reference it as needed vs trying to invent from wholecloth via diffusion. Object persistence!
Kind of makes me think that autonomous vehicles will eventually gather better dataset (although limited to roads!), since they do have 360 degree cameras and lidar, so they can track objects before & after they are shown on certain view.
The model already ingests text, so you just to attach a tiny VLM head that reviews the image and read / writes scene continuity text tokens.
Once you have that you can probably just RL with scoring on continuity issues and eventually the VLM head will learn to update dense continuity tokens?
So many fun wierd experiments to run if you have some compute to mess with. There are competent open video diffusion models that you could test this concept easily enough.