Hacker News Reader

Possibly less sophisticatedly I think of them as a sandwich of affine maps and nonlinear isotropies (as those giving irregular rings in tree trunks). The affinities are represented nicely in GL(n+1) with a homogenous coordinates trick related to neuron biases. A question would be if there's something interesting to say about the interactions of the affinities and isotropies in group theoretic terms (which I dunno).

View on HN

ReLU is very simple in this regard. In the plain form it's just affine transformation followed by 'a viewport'. The mapping trough multiple layers is alternating affine transformations and windows into data. Learning is combination of squeezing and rotating data so that it can be seen or unseen from the window and rotating window frames to do the same.

Any extra stuff, like batch normalization between the layers can again introduce more complex nonlinearity.

Really agree with the extra simplicity in the ReLU case. Liwen Zhang, Gregory Naitzat and Lek-Heng Lim showed last year that "the family of such neural networks is equivalent to the family of tropical rational maps", where rational functions are quotient of polynomials and "tropical" is in the context of Tropical Geometry, where instead of the typical "plus, times" ring one uses a "min, plus" ring that has somewhat unexpected applications. For instance, with its module theory one can calculate minimum costs paths in graphs just as one does for reachability starting with the adjacency matrix of a graph over a boolean semiring. arXiv:1805.07091v1

Reply on news.ycombinator.com