Hacker News Reader

Comment by friendzis | Hacker News Reader

A bit out of topic, but has there been any research done on learn-time topology optimisations? E.g. Start with 4-4-4-4 network, remove some little contributing nodes/connections during training, resulting in 4-3-2-4 network with similar accuracy as 4x4, but higher performance?

Intuitively I think that there should be connections activated for less than x% of inputs that could be removed entirely. In some cases this would mean removal of whole node. Would interesting to read something about such an approach.

View on HN

That's a basic part of regularization! Weights are "penalized" for being too big, and encouraged to be as small as possible. This results in most of the weights being zero, which is the same as them not existing at all.

There is some research on pruning networks after training, removing small weights and nodes that don't contribute much. This results in much smaller neural networks, which run better on smartphones or embedded devices.

This isn't done often though because on GPUs it doesn't result in higher performance. Because of the way SIMD instructions work or something, there's no way to take advantage of sparser neural networks. A synapse of weight 0 still requires a multiplication by zero, and adding the zero to the sum.

TWEANNS - Topology and Weight Evolving Neural Networks. Gene Sher's book 'The Handbook of Neuroevolution Through Erlang' presents them, and codes them in Erlang. People have coded them in Elixir, and I am currently trying in LFE (Lisp Flavored Erlang). They can add neurons and connections or take them away based upon a GA approach (Genetic Algorithms). A big book, but that is because it covers the material thoroughly, and with lots of explanations. I have been reading NN books since the early 90s, and this one would have been a great format back then.

I don't think this has been tried. What would be the heuristic for removing connections/nodes? It does seem to me, though, that the heuristic should be "learn from the data using backprop" and that you're essentially calling for L1 regularization (which encourages sparsity), or L2 regularization (which encourages small weights), which are already pretty standard in machine learning.

Something like this?

http://arxiv.org/abs/1603.09382v1

Stochastic depth is just a form of regularization over layers, it doesn't save any parameters or remove any connections.

rtNEAT / NEAT is vaguely like this. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.28....

Reply on news.ycombinator.com