Hacker News Reader

Comment by hprotagonist | Hacker News Reader

I have been writing optimization solvers of many forms to solve problems in engineering for about 20 years. from "make excel do linear regression on some data" to linear least squares to some nonlinear methods, simulated annealing, bayesian methods, deep neural networks -- none of this is "a new kind of programming", it's "do a bunch of data munging, throw matrix at a function, get matrix back, interpret/plot."

there's really no other magic in it than that.

View on HN

How well does "linear least squares to some nonlinear methods, simulated annealing, bayesian methods" work when you're doing speech to text, text generation or object detection? Deep learning is different in that it's a huge leap closer to human capabilities compared to the other methods you've listed.

Deep learning is closer to linear regression than it’s to human capabilities.

> Deep learning is closer to linear regression than it’s to human capabilities.

Quantitatively perhaps, in that deep learning is equivalent to many neurons, whereas a least squares model is only equivalent to a single neuron. Other than that there is not much evidence that a human brain is qualitatively different.

Deep learning is much more than a "long" network nowadays. Most of DL success comes from clever architectures, like Inception Net. AFAIK there's no evidence human brain is anything like those fancy deep NNs.

well, your neural network is just a very complicated - ideally continously differentiable - function. In principle you could just fit this function to data with nonlinear least squares (e.g. Levenberg-Marquardt), we use backpropagation and SGD, because it's faster. Concerning simulated annealing: have a look at the old Alphafold and check how they optimize their neural networks...

So, could you explain again how the concepts and foundations of deep learning differs from plain old regression techniques?

Recurrent networks don't give you a matrix back. They return a state machine. Qualitatively different.

Transitioning from a function that returns a scalar, to returning a polynomial, to returning an arbitrary function, and now to returning something stateful is a big difference. I suppose the next step is to write an algorithm that trains a Turing machine.

RNNs provided by common frameworks really do give a matrix back (examples [0, 1]). They're about as stateful as any other object or generator function.

[0]: https://pytorch.org/docs/stable/generated/torch.nn.RNN.html

[1]: https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cu...

What other statistical machine learning algorithm gives you a state machine as an output?

The particular encoding -- matrix or otherwise -- is just an implementation detail.

> What other statistical machine learning algorithm gives you a state machine as an output?

* hidden Markov models

* autoregressive models

* learned LQR

Sure, some of those have finite memory, but RNNs are practically limited–not as easily quantified.

I see what you're saying about the single-step operation appearing unique, but I also think that RNNs can be viewed through a lens such that they look like "normal" programming concepts like generators and folding iterators.

I'm not familiar with LQRs. Versus HMMs and linear autoregressive models, I think avoiding the need to predefine the form of the autoregression is a bigger leap than it might appear right now. I'd always been a neural network skeptic, thinking they're excessively complex for no benefit. It seems to me that's no longer true. I'm not certain that'll hold, but it's exciting.

The feedback machine is also represented as a matrix, with either implicit or explicit recurrence by the code around the model.

Yes, a finite state machine can be encoded as a matrix. Ultimately it's just a string of bits, represented by whatever physical mechanism it happens to reside in. It's still a state machine, and that's a big difference.

Reply on news.ycombinator.com