Looks like, again, this doesn't have GPT-4 processing video as much as a stack of video frames, concatenated and sent as a single image. But much closer to real!
And worse: turns out GPT-4 isn't processing pixels so much as integers representing in a position in some color space like RGB!
And worse! turns out GPT-4 isn't processing integers so much as series of ones and zeroes!
Now that this is public knowledge, I'm willing to bet this was the ugly "less than candid" truth that the board sacked Sam Altman over.
Using captions to bridge this only works to a certain extent (you're giving text descriptions of what happened in the past, not what had _actually_ happened).
On top of that the video container usually provides synchronized audio packets.