Hacker News Reader

Comment by ganfortran | Hacker News Reader

> deep learning-based solutions lack mathematical elegance and offer very little interpretability of the found solution or understanding of the underlying phenomena

I disagree. Elegance and interpretability comes with one big flaw: assumption, lots of them. While deep learning based methods assume few or not. If they are so elegant why would them underperform? Probably it is because the problem we are trying to solve doesn't follow the assumptions, like convexity, etc.

In that sense, one can say deep learning methods are even more elegant, because they can work end-2-end. In that sense, NN can be a blessing, because now we got the answer, we don't need to assume anything, but just to DECODE it.

View on HN

There is a no free lunch theorem in learning, which says that without limiting your hypothesis space, you cannot learn. That means, you need to have some assumptions.

No doubt that DNNs are subject to the same theorem, the only question is, how does their hypothesis space looks like? Does anyone have idea about this? I suspect we don't really know what the DNNs assumptions are.

The DNNs assumptions are "The true model can be fit by this network architecture". If you use convolutions, then an additional assumption is translation invariance.

> The DNNs assumptions are "The true model can be fit by this network architecture"

Yeah, but that's tautological. What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn? Or more importantly, what functions it cannot learn? Or at least, how many bits are assumed and how many bits have to be learned?

I think that is what bothers people about DNNs. I personally think they are sometimes even inefficient - we are learning them much more parameters (bits) than the actual hypothesis space requires.

> What I mean can somebody sit down and for a given DNN architecture, write down (at least approximately) the set of functions that it can learn?

For a two-layer architecture with ReLU activations and n units in the hidden layer, this is the set of piecewise linear continuous functions with n kinks.

How many of those do you need to tell the difference between cats and dogs in random internet pictures?

Agree with you. I think what I need to make it more clear, is DNN puts little assumption on data. It usually consumes it as raw pixels, in case of images.

The architecture of NN itself is the biggest assumption here, as it is how it will be used to process the data. But they those operators are usually generic, a lot of them just umbrella operators for matrix multiplications + non-linearity in between.

I wrote a bit about the theoretical reasons DNNs work here: http://lesswrong.com/lw/m9p/approximating_solomonoff_inducti... There's some more interesting discussion in the comments there.

Well here's an argument for interpretability: how do you know when your model will fail?

One tradeoff might be that with deep learning, it's unlikely to fail, but you won't know when it will. With conventional methods, you may not get as good of a performance in comparison, but have a better sense of the domain of applicability because you have a better understanding of the feature space (for instance, you can determine similarity of new data in this feature space when it arrives to determine whether it is similar to past data your model has been trained on).

You can do the same with deep learning, your model will fail when the train data and test data are systematically different. This isn't about understanding the model, it is about understanding the data. Understanding the data is much more important than interpreting the model. Systematic uncertainty is a problem for all models.

Yes - but how do you know when your test data and training data are different in the way that the model cares? With conventional methods, you can assess similarity by distances in the feature space of the model, or you have a physical understanding of why the model works than you have a better intuition about the differences in data that are important.

> because they can work end-2-end

Agree. Not just end-2-end, but the same model can be applied to many different tasks. DeepMind's recent papers make progress in image generation, audio generation, language modeling & machine translation using the same basic construct of a masked dilated shifted convolutions.

Reply on news.ycombinator.com