This is mad impressive, for sure. With most AI problems, I can at least comprehend the approach, and the necessary combination of models. Not so much here.

My first question is what is the input to the AI? Is it the raw pixel array of the display? Or does it get API-level readouts of what’s happening? Because implementing the CV just to segment the display output in real time is crazy enough. I would assume the latter.

I think this basically proves that any problem that can be exhaustively simulated is solvable now. This may mark a tipping point, as every problem for which simulations exist (essentially infinite labels) is solved - then the balance will tip back toward making faster and more accurate sims (think multi-scale first principles physics stuff).