You'll find pretty quickly that the exact techniques that worked so well in learning to play Atari games very often fail spectacularly when you have to introduce goal-steering.

Reinforcement learning turns out to be fantastically clever at finding really stupid solutions if you give it the tiniest opening to do so. You put a camera in a room to provide feedback for an agent to learn to move an object closer to the camera, and it will happily learn to knock the camera over.