The main result is the local Hebbian-like learning converges to exactly the same gradients as the ones produced with backprop.
"In terms of computational cost, one inference iteration in the predictive coding network is about as costly as a backprop backwards pass. Thus, due to using 100-200 iterations for full convergence, our algorithm is substantially more expensive than backprop which limits the scalability of our method. However, this serial cost is misleading when talking about highly parallel neural architectures. In the brain, neurons cannot wait for a sequential forward and backward sweep. By phrasing our algorithm as a global descent, our algorithm is fully parallel across layers. There is no waiting and no phases to be coordinated. Each neuron need only respond to its local driving inputs and downwards error signals. We believe that this local and parallelizable property of our algorithm may engender the possibility of substantially more efficient implementations on neuromorphic hardware"
The main idea is based on [1]: a brain can learn exactly the same things as what can be learned with backprop. How come this is not more widely known? This sounds like a huge breakthrough to me.
https://www.mitpressjournals.org/doi/full/10.1162/NECO_a_009...
- Hochreiter and Schmidhuber present a truncated version of BPTT in the original LSTM paper that tracks part of the error locally and forward in time (https://www.bioinf.jku.at/publications/older/2604.pdf)
- Bellec et al. https://www.nature.com/articles/s41467-020-17236-y present a similar algorithm that tracks the error with something they call eligibility traces. Despite the title of the paper the method is also applicable to non-spiking RNNs.
- Bohnstigl et al. present another solution: https://arxiv.org/pdf/2007.12723.pdf
Even on the potential neuromophic hardware it's "only" parallel with the number of layers. That's useful but not as parallel as current approaches.
I think that neurons hate when we animate them.
Being a little more serious, is it possible to perform gradual back propagation? I.e., for iteration 1 train only last layer, for iteration 2 change two last layers, for iteration 4 train three last layers, etc.
I also have to say that in my spare time I am investigating a way to train networks on the whole set as opposed to the single sample or batch back propagation and one result I've got so far is not quite uninteresting, in my opinion: https://github.com/thesz/higgs-logistic-regression
The approximation of gradients is still use of gradients. It still computes updates which are need to be shared and still has some bottleneck on scalability.