They don’t use them indeed. I was replying to the general idea about the additions.
OTOH Gaussian kernels smoothen almost everything. Maybe it will be stable even with sin(1/x) as an “activation”.
OTOH Gaussian kernels smoothen almost everything. Maybe it will be stable even with sin(1/x) as an “activation”.
Recalling the definition of exact differentiability is irrelevant.
Instead take the smallest interval that you can represent in fp32 not too far away from zero for example. Take few values in that infinite interval and check the behaviour of the said monstrous function.
This is a “trivial” example when studying eg. Distribution theory.
Said differently, you need to assess how smooth is the differential operator itself.