by trying to learning the general rules that to explain the dataset and minimise its loss. That's what machine learning is about, it's not called machine memorising.
The problem is not a failure to understand the rules. It is just not very good at maintaining the state.