This might sound unrealistically complicated, and the approach suggested here certainly is, but the idea of simulating a world is a very important concept in our current models of human intelligence, with very small children spending a lot of their time learning basic concepts of cause and effect, properties of objects, materials and physics and so on.

It reminds me of something I do in my current occupation, with a robot. It makes predictions about how a certain motor movement will result in a change in position, then measures progress in the real world. Should the prediction and reality diverge significantly (more than a particular amount), it stops and calls a human to look into what happened. I can imagine AI techniques using this kind of thing to guess what will happen next in a series of events, with the result's divergance from the guess weighting the learning rather than a steadily decreasing weight. Perhaps this is already a thing though, I'm no AI researcher