This is also a huge problem in offline Rl (learning a policy using only a dataset). If done naively, the learned policy will keep accumulating errors due to enter areas that are not well covered. So the trick is to avoid these areas. In offline rl they do this by measuring epistemic uncertainty and using this as a regularization term in the loss function such that the model learns to avoid these areas. This a good blog post that explains it way better https://jacobbuckman.com/2020-11-30-conceptual-fundamentals-...