Real-time prompt-based stream editing for AR at 1080p, 46 FPS and 87 ms latency

Author here. This is a report about my latest hacking experience: a high-speed stream-editing pipeline for AR. It received Best Demo honorable mention at IEEE ISMAR 2026.

The system accepts a live camera feed, which is modified in real-time according to a text prompt. The result is presented on a large screen (a magic mirror) and has interactive speeds. It has about 5x the pixel throughput of StreamDiffusionV2 at a quarter of its latency, on two GPUs where it uses four.

I developed the demo all by myself (unusual for a professor) with agent swarms doing the implementation.

The development was made possible by 3 key ingredients:

- Heavy use of Karpathy's autoresearch methodology on several levels of abstraction

- Having access to a nice GPU cluster (~1000 GB200 hours used)

- Open weight models: a model 10x slower, such as Claude Fable, would not have been applicable for my main autoresearch loop, which ran for about 1 month.

The end result is a demo that shows the emerging possibility of using diffusion transformers for AR and VR systems, instead of Gaussian Splatting that so many people bet on.

There are still things I need to figure out:

- How to improve temporal consistency

- How to make edits robust for multiple humans in the frame

I learned several things, including:

- Autoresearch is unreasonably effective.

- 1000 GB200 GPU hours were wasteful. I should have planned better.

- Agent swarms on multiple heterogeneous machines are not trivial

- 87 ms is too high for deployment on AR Glasses. I need to find a way to bring it down to 20 ms

- The numerous bugs of TensorRT on Blackwell, e.g. failures for high resolution VAE encode/decode