I built a harness that lets Claude Code play Clash Royale autonomously through screen capture and input automation. The AI makes decisions from raw screenshots and executes them via shell scripts that click screen coordinates.
The core challenge was latency. Each cycle (screenshot → vision analysis → decision → tool call) takes ~7 seconds. Far too slow for a real-time game. My solution: spawn 3 parallel agents playing the same match simultaneously. Their natural stagger produces a card play every 2-3 seconds. Brute force, but effective.
Architecture is a Commander agent managing menus and game flow, spawning Player sub-agents for each match. The coordinate system maps UI elements and a placement grid so agents can call play_card.sh B2 giant and have it translate to pixel coordinates.
Results: 1000+ trophies (Arena 5). Below 1000 you're mostly playing bots - crossing that threshold means real players with better cards and actual strategy. Latency becomes punishing when humans can exploit slow reactions.
Interesting findings:
Claude researched its own deck strategy and wrote the sub-agent gameplay prompts
The game shows "WINNER!" on every result screen including losses - had to teach it to verify via trophy delta
Added an auto-opener (first card at 5 sec) because Claude was too slow to establish early tempo
Streamed 12+ hours on Twitch. Claude played 5 consecutive games with minimal human intervention (just closing pop-ups and opening chests).
To my knowledge this is the first harness for frontier AI playing Clash Royale. Potentially interesting as a benchmark - tests vision, real-time decision making, resource management, and multi-agent coordination.