Hacker News Reader

Comment by xapata | Hacker News Reader

Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different.

Transitioning from a function that returns a scalar, to returning a polynomial, to returning an arbitrary function, and now to returning something stateful is a big difference. I suppose the next step is to write an algorithm that trains a Turing machine.

View on HN

RNNs provided by common frameworks really do give a matrix back (examples [0, 1]). They're about as stateful as any other object or generator function.

[0]: https://pytorch.org/docs/stable/generated/torch.nn.RNN.html

[1]: https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cu...

What other statistical machine learning algorithm gives you a state machine as an output?

The particular encoding -- matrix or otherwise -- is just an implementation detail.

> What other statistical machine learning algorithm gives you a state machine as an output?

* hidden Markov models

* autoregressive models

* learned LQR

Sure, some of those have finite memory, but RNNs are practically limited–not as easily quantified.

I see what you're saying about the single-step operation appearing unique, but I also think that RNNs can be viewed through a lens such that they look like "normal" programming concepts like generators and folding iterators.

I'm not familiar with LQRs. Versus HMMs and linear autoregressive models, I think avoiding the need to predefine the form of the autoregression is a bigger leap than it might appear right now. I'd always been a neural network skeptic, thinking they're excessively complex for no benefit. It seems to me that's no longer true. I'm not certain that'll hold, but it's exciting.

The feedback machine is also represented as a matrix, with either implicit or explicit recurrence by the code around the model.

Yes, a finite state machine can be encoded as a matrix. Ultimately it's just a string of bits, represented by whatever physical mechanism it happens to reside in. It's still a state machine, and that's a big difference.

Reply on news.ycombinator.com