Hacker News Reader

Comment by pmarreck | Hacker News Reader

I guess the obvious next step is to cull the dead neurons from the model data? I imagine the brain does the same thing after the childhood years

View on HN

Most models have a purely fixed architecture: i.e. if you train three layers of six feed-forward neurons a piece, your model and the training data it learns from will just fit within that architecture to the extent that gradient descent can force it to fit. There is no mechanism to say "oh these neurons will never activate, let's prune them", or "we'd have much better loss if we added a layer here".

In the dark times before PyTorch there was an idea called NEAT: "neuroevolution of augmenting topologies", which tried to use a genetic algorithm (i.e. testing a bunch of slightly modified solutions for loss) to discover both optimal weights and optimal network structure. I don't think this gets used all that often, if at all.

I hear stories every few years about how many models have unused neurons that can be pruned or how hyperparameter selection is a pain in the ass, but nothing about automating the pruning and parameter selection in such a way as to efficiently use the whole model. I'm not sure it's necessary anyway.

DL researcher here, it's also hard I think because experimentally many of us have noted (also some research on this) that there's a critical early phase to learning that conditions the network a certain way, and adding and/or removing layers later seems to actually be quite difficult (removing easier than adding, by far, esp w/ something like variational dropout [yes, i cite the old deep magicks]: https://arxiv.org/abs/1701.05369)

I had success doubling the width of a layer by copying, adding epsilon noise and then fine tuning.

Transform from [X] to [X X+epsilon].

In that case I was trying to fuse two signals that had fairly similar characteristics.

Yes, unfortunately I have yet to beat that technique in wallclock convergence speed vs just using the larger network from the start. :'((((

Whoever figures out how to clearly and effectively do it consistently faster than a 'full network from the start' version will open up an entirely new subfield of neural network training efficiency. :'))))

There's a huge amount of work on model pruning, especially with an eye towards model reduction for on-device deployments. I've done a bunch of work in this space focused on getting speech synthesis working in real time on phones. It works and can be automated.

There's a lot of nuance though. What has typically worked best are variants on iterative magnitude weight pruning rather than activation pruning.

This can often get rid of 90% of the weights with minimal impact on quality. Structured pruning lets you remove blocks of weights while retaining good SIMD utilization... and so on.

> There is no mechanism to say "oh these neurons will never activate, let's prune them"

Not at training time. But after training you can evaluate which activations rarely fire and simply remove their columns from the weight matrix.

Or you could re-randomize them and continue training.

A neuron which rarely fires may be responding to a rare feature, no?

Absolutely. Pruning can decrease overall accuracy. The challenge is to prune as aggressively as possible while keeping accuracy as high as possible. It's certainly not an operation that is expected to have no effect.

It should be noted that there are methods for training the network to encourage these prunable neurons, eg sparsity penalties.

There are already methods to do that in neural networks, it's called "pruning"

https://en.wikipedia.org/wiki/Pruning_(artificial_neural_net...

and, yes the article also mentions that it happens in mammal brains.

Reply on news.ycombinator.com