In a simulation environment you know ground truth. (Where objects actually were vs. what you saw.) You can play the same scenario from slightly different angles. Or different conditions (rain, snow, low light, bright light). You can look at counterfactuals (what would I have seen if I had done X). You can test the behavior of your system without actually setting a computer loose in the real world.

From "real world" recorded data you have none of these things.

This is the difference between observational and experimental data and it's a huge difference.