Proper initialization of layers keeps gradient magnitudes from vanishing/exploding in deep networks. If you make sure the output of each layer has mean 0, std 1, the gradients will be reasonable as well, for example.

I recommend e.g. the og resnet paper and its follow-up from Kaiming He et al.

For a modern take on RNNs, read https://arxiv.org/abs/2303.06349 by DeepMind.

There essentially the point is that largest eigenvalue (spectral radius) needs to be around 1, meaning repeated applications of a linear transformation doesn’t cause increase or decrease of the activations.