Yeah, pretty much. It might continue to work fine with very minor changes, but as we know how to design and build them today deep neural networks are often very sensitive to minor changes in the input distribution. The key insight here though is not about deep nets, but about our true progress towards AGI. These systems don't hypothesize about the best strategy to take, think about different approaches, formulate alternatives, etc. They basically memorize paths that have worked based on random exploration. We have what seems like a long way to go (nobody can know whether we are 1 or 100 key ideas away) before an AI already trained to play 5 Atari games can be turned towards a new game and play very well based on its experience with related genres. Today they are trained from scratch for each game, so although the model might be the same the AI is not able to transfer theories and strategies from one domain to another.

Note you can create a game where the simulated world has random variations and the RL algorithm will learn to handle it. If you don't train for it, obviously it won't learn it.

Check out this video from mit/openai: https://www.youtube.com/watch?v=9EN_HoEk3KY The entire talk is interesting but the section at 21:40 talks about "Sim2Real with Meta Learning".