[tangential to this really cool demo] JPEG images being the only possible interface to GPT-4 feels wasteful. the human eye works the delta between "frames", not the image itself. I wonder if the next big step that would allow real-time video processing at high resolutions is to have the model's internal state operate on keyframes and deltas similar to how video codecs like MPEG work.