There would be a reality gap, going from simulation to real world. Good read: https://ai.googleblog.com/2017/10/closing-simulation-to-real...