Pretty cool. I wonder if there's enough interest from other devs to hook into something like this or build "apps" on top of this ability. Are you trying to "productionize" or build a platform for others on this or was it a cool hack that you don't plan on spending much more time on? Did you send screenshots at a certain FPS or do some other decision making on when to use the bigger LLM?

My little startup has system that can stream apps to an ML processing service with both a video of what's on screen and things like the context around it and what you clicked on. We run an LLM on top of this after a bunch of other processing (OCR, delta change detection, speech reco, etc.) for our knowledge capture purposes for our single app. It would be really straight forward to make such a platform available to others to build apps like what you're showing on more then just a browser, pretty much any application you run on a desktop or working across multiple of them, we haven't since before the new GPT4 with vision, most people weren't working on anything where that would help.

Anyway, I think we've solved a bunch of the heavy lifting to make this possible, feel free to email me, or anyone else reading this who like this space, if you're a company or dev that might want that layer so you can build something cool on top of it like this (diamond@augmend.com).

Can you share the script? It’s a great idea

Yes I feel like the delay seen in the video is about the same length as I would expect when making a request without streaming.