HN: how do YOU think about NNs? As a matter of pure preference, most of my daydreaming of NNs comes from my inductive biases of (1) and (2), with a few ideas from MCMC methods and optimization thrown in the mix.
> Representation theory
One question I've always wondered is whether it is better to get a network to (1) learn representation transformations A->B->C->D via a series of layers, or (2) try to learn A->D via a single (highly nonlinear) transformation.
Obviously the former lends itself better to analytical understanding, and this seems to be what emerges when you train ImageNet. However, do researchers have any control over whether the learned net does (1) or (2)? I'm thinking that scattering "skip arcs" that pass gradients through layers (or an LSTM) would cause representations to mix more between layers.
> Each layer is a function, acting on the output of a previous layer. As a whole, the network is a chain of composed functions. This chain of composed functions is optimized to perform a task.
This is certainly true for the majority of models in use today, but I wonder if this will remain true for future architectures. In practice, most layer types store some state, and implement "forward" and "backward" functions that may utilize that state.
One could argue that state is not necessary if we only consider the network to be defined by its feedforward passes (de-coupled from the optimization problem, which requires the "backward" functions). But the neuroscience narrative argues that function and learning cannot be decoupled.
In the case of a network that is supposed to update it's own weights during normal operation (i.e. topic discovery), the functional narrative is less clear to me. Thoughts?