> We built the text side of it first, and the text side is definitely... more powerful. How these models can reason about images is getting better, but I think it's a decent bit behind.
This seems to be the main issue: using an AI model predominantly trained on text-based reasoning to play through a graphical video game challenge. Given this context, the image processing for this model is like an underdeveloped skill compared to its text-based reasoning. Even though it spent an excessive amount of time navigating through Mt. Moon or getting trapped in small areas of the map, Claude will likely only get better at playing Pokemon or other games as it's trained on more image-related tasks and the model balances out its capabilities.