Comment by stiff | Hacker News Reader

Comment by stiff | Hacker News Reader

Parent Full thread

I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside).

There are many ways to avoid this, for example have a look at:

http://en.wikipedia.org/wiki/Rprop http://en.wikipedia.org/wiki/Conjugate_gradient_method

In traditional (I am not talking about the "deep" stuff) neural networks optimization is hardly ever the problem though, most often under- or over- fitting is the issue that produces poor performance.

One of the performance problems with neural networks is that the number of cores on a typical machine are far less than the number of input and intermediate nodes in the network.

This sounds odd. Certainly doing neural networks in hardware is interesting, but it sounds a bit like an imagined problem, I mean, when multiplying 20 numbers one does not complain that the number of cores is less than 20. And most often it is the training of the network that is resource-intensive not the actual running of it.

Both RProp and Conjugate Gradient are descent methods-- they find a path to a local minima from the starting configuration. They do not help with finding the global minimum.

Using simulated annealing to guide random initializations can help find a better minima, but to get the global minima with simulated annealing takes an inordinate amount of time.

(Poor) local minima can be sometimes reached in back propagation due to a bad setting of the learning rate and RProp does help with that. You can also repeat the learning process using one of those algorithms many times with different initial random weights and use the results of the best attempt. I think this most often suffices in practical applications of NN for getting decent results, you are right I got some of this wrong, I basically wanted to say that backprop is no longer state of the art in training and that optimization is the relatively easy part of learning.

Well, if you've got 20 numbers to multiply you'll get the job done fastest if you do them in parallel with 20 dedicated multipliers.

There's an obvious vision of building a "neural circuit" where there is some specialized processor for each neuron but my guess is that it gets difficult when you consider the communication fabric required between the layers.

The first artificial neural networks were actually done in hardware, here is Frank Rosenblatt with his Perceptron:

http://www.enzyklopaedie-der-wirtschaftsinformatik.de/wi-enz...

As said, it is an interesting thing to consider as an alternative computer architecture and what not, but I just have some doubts if people practically using neural networks right now really run into performance problems because of having too little cores. I don't think this is more true for neural networks than for anything else.

Reply on news.ycombinator.com