So this is something that can be somewhat explained using not terribly handwavy mathematics. Picking hyperparameters on the other hand...
So this is something that can be somewhat explained using not terribly handwavy mathematics. Picking hyperparameters on the other hand...
Pick eg. x -> sin(1/x) around zero and its derivatives.
The small modifications that you’re talking about are on the argument. These can lead to huuge changes in the values.
The stability is more likely due to the diffusive nature of the models and well executed trainings.
OTOH Gaussian kernels smoothen almost everything. Maybe it will be stable even with sin(1/x) as an “activation”.
Recalling the definition of exact differentiability is irrelevant.
Instead take the smallest interval that you can represent in fp32 not too far away from zero for example. Take few values in that infinite interval and check the behaviour of the said monstrous function.
This is a “trivial” example when studying eg. Distribution theory.
Said differently, you need to assess how smooth is the differential operator itself.