Hacker News Reader

Comment by j2kun | Hacker News Reader

Can someone explain to me why NN's are always in layers?

From the first chapter of this book, it is described that each layer adds more abstraction. The first layer (input) works on individual pixels, the next layer on a part of the image. Each deeper layer can make more high-level decissions.

I understand why more layers adds complexity, but that doesn't answer my question: why layers instead of an arbitrary acyclic graph?

I'm not an expert, but my guess is because it makes it really easy to materialize the weights between layers as a 2d matrix and we have some really good library code out there for dealing with matrices.

There's probably room for experimenting with more novel constructions and even some papers out there looking into the matter.

Efficiency is the expected answer. I'm just wondering if there's a more theoretical reason, such as "every function that can be computed by a non-layered acyclic network can be computed by a complete layered network using only a small number of extra nodes/layers."

I think that it can. With some weights of 0 and some weights of 1, you can trivially map 'jumps' that skip from a node in one layer to a node a couple layers distant, by means of some incorporate-no-other-inputs intermediate nodes, right? Sigmoid function on 1 is still 1? Once you have those, it's just a matter of how many layers you need for any acyclic structure, I think.

Although if you wanted to come up with difficult scenarios, it's not hard to think of structures that would make some of those middle layers really tall, or add a lot of middle layers.

As I mentioned in another branch of this thread, selectively choosing edges between nodes isn't an option, because in the standard model you have complete incidence between nodes in adjacent layers.

Feedforward neural networks are acyclic; recurrent neural networks are cyclic. Recurrent NNs are harder to train in general.

Layered is much more specific than acyclic. I can come up with an example that is not layered by still acyclic. Just connect nodes across two or more layers.

Functionally, layered and acyclic NNs are the same thing. An arbitrary acyclic NN acts the same as a layered NN with some of the weights fixed at 1 or 0. (Replace any edge crossing n layers in a non-layered acyclic NN with a chain of n nodes to get a layered NN that responds to input the same as the original non-layered NN.)

I suppose there may be some cases where the extra speed you get by omitting the intermediate nodes pays off. However, I can't imagine a situation where you'd know enough about the problem in advance that you could design the NN's graph in that level of detail.

Usually the adjacency between layers is complete, so your modification isn't quite without loss of generality.

I also cannot imagine a situation in which you'd know that much detail, but using a general network would allow one to, for example, dynamically modify the topology of the graph (as real neural networks do regularly).

EDIT: I guess what I'm asking for is a rigorous proof that the two models are equivalent with as little overhead as you say there should be.

Reply on news.ycombinator.com