I think this is referring to inaccuracies of simulation which the RL model can learn to exploit. Sort of a local minimum that doesn't exist in the real world.
Say you're teaching a robot to walk, but the joint constraints in the simulator aren't quite right. It would be possible for the policy to exert more torque on the joint than it's supposed to be able to in real life. Take that policy and put it on a real robot and it won't work.