Why Do Neural Networks Need an Activation Function?
datastuff.tech
datastuff.tech
Huh?
Backpropagation is essentially a big chain rule application. If you order the layers into a list (with identity transformations to pipe the values that do not change in the particular layer) then the whole process is just multiplying the derivative matrices, starting with the loss function and then the bottom-most layer. That's why it's difficult to train long unrolled sequences of RNNs - the many matrices that you have to multiply on the way back are either contracting (in which case the signal dies out) or expanding (in which case you get NaNs). That's also why people "cheat" backpropagation by using things like gradient clipping.
https://medium.com/@karpathy/yes-you-should-understand-backp...
Here's an attempt to make the backprop linear: https://openreview.net/forum?id=ByfPDyrYim
Anyhow we're talking past each other. I mean that it's linear in a very strict/specific sense. Take a fixed newtork with a fixed loss function, for a particular set of inputs and particular target outputs (think 1 batch, or even just one example). Now as you backprop you continuously compute derivatives of what's currently on the layer with respect to the loss value, starting from the loss function, then the bottom most layer, then the previous layer and so on.
Now this can be decomposed into a series of matrix multiplication (chain rule), where yes, in any particular run the matrices depend on the weights, inputs and outputs in a nonlinear fashion. But the overall effect is just a composition of linear operators (matrix multiplication). That's why things go to shit when the network is deep and the derivative matrices are all contracting or all expanding.
And just to connect with your example: if f=d(error)/dW then yes d(2error)/dW=2f.