I don't recall SD or variants using discontinuous terms like 1/x. Sigmoid, softmax, and SiLU are going to be what you're looking for.
OTOH Gaussian kernels smoothen almost everything. Maybe it will be stable even with sin(1/x) as an “activation”.
Recalling the definition of exact differentiability is irrelevant.
Instead take the smallest interval that you can represent in fp32 not too far away from zero for example. Take few values in that infinite interval and check the behaviour of the said monstrous function.
This is a “trivial” example when studying eg. Distribution theory.
Said differently, you need to assess how smooth is the differential operator itself.