Sounds like a nice proof of concept, but I would have tried to identify the in-game variables as in [0] and identify the state of the game from there. Maybe it would take a bit longer, but in the end one could track if a player is sliding or was using an item and so on. Grabbing video frames and doing some kind of template matching on the still images sounds a bit... inconvenient.