One of the things about continuous functions is that they aren't necessarily differentiable, cf. https://en.wikipedia.org/wiki/Weierstrass_function
> A neural network computes linear transformations
They compute a nonlinear function of an affine transform. Neural networks are nonlinear functions.