Pick eg. x -> sin(1/x) around zero and its derivatives.
The small modifications that you’re talking about are on the argument. These can lead to huuge changes in the values.
The stability is more likely due to the diffusive nature of the models and well executed trainings.