Peering into neural networks | Hacker News Reader

Peering into neural networks

news.mit.edu

79 pointsravenkat19 comments

1.) For the umpteenth time, they're not black boxes. We can inspect everything in the structure.

2.) "a team of computer-vision researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL)" may have "described a method for peering into" the not-black box of a convnet two years ago, but Oxford researchers published on in in 2013.

3.) Gushing about how understanding convolutional networks can help confirm the grandmother cell hypothesis in real brains is embarrassing under all circumstances but should be particularly so when thorough examinations from real brains just came out to the considerable detriment of said hypothesis. http://www.cell.com/cell/fulltext/S0092-8674(17)30538-X

Nothing wrong with making visualizations of your nets, but I'm less than impressed by the reporting.

Few things are black boxes in absolute terms. It's the difficulty of understanding NN's that makes them black boxes. Personally I think the black box analogy is accurate for any net of appreciable size.

I think your point about the frequently gradual nature of the concept is a very good one, and I'll buy that there may be appropriate analogies to a black box in this context, especially if some sort of poetic or metaphoric description is what one is aiming for. I see no indication of that in the article. What the present journalists, and many others, are doing in this respect is not analogy but rather destructive appropriation of terminology.

I get that they mean something like 'is difficult to understand' too, which is in many ways completely uncontroversial. (I'm certainly not going to claim it's a solved problem to quickly and effectively come understand intellectually how an arbitrary ANN does what it does. I doubt there are many people who would.) If that's what they mean, then they can say that, or make up their own picture language that isn't already busy meaning the polar opposite of the situation they're alluding to. A black box is characterized by observability only at the edges and unknown inner workings. It is by definition an inappropriate term for a convolutional network where every single weight and operation and intermediate result is trivially inspectable and you can do things like follow the effects of an experimental perturbation along every step of every path through the network. 'But it's really hard to get an intuitive understanding of what that means in the big picture' is a perfectly legitimately concern for something to improve on, but it isn't remotely good enough as an excuse for effectively claiming we have no observability or control where both are clearly abundant.

Abusing the term like this detracts from its established role in engineering, systems theory, etcetera and in my opinion also from communicating the actual problems of understanding how ANN:s do their thing. I really wish they'd stop making that claim.

Now I'm going to leave my computer before I get started on the ######s talking about steep learning curves as if they were an obstacle.

On (1), this is a figure of speech. Closed source software, especially without clear ICDs, is often referred to as a black box even though we can almost always fully examine it, to every binary byte.

This particular example for visualizing neural networks doesn't seem particularly enlightening but I would argue the "black box" term is a appropriate and established term for describing current neural networks. In fact, researchers in the field acknowledge the problem of prediction without clear explanation and sometimes use this term to the describe the difficulty involved in reasoning about neural networks - it's some journalistic simplification or urban myth.

For example, "Many recent advances in NLP problems have come from formulating and training expressive and elabo- rate neural models. This includes models for senti- ment classification, parsing, and machine translation among many others. The gains in accuracy have, however, come at the cost of interpretability since complex neural models offer little transparency con- cerning their inner workings."

Rationalizing Neural Predictions, Tao Lei, Regina Barzilay and Tommi Jaakkola

Also: "When people say, "Neural networks are black boxes", what they mean is that it is hard to look "into" the network and figure out exactly what it has learnt.

"In a hand-crafted pipeline, you know precisely what you are building. So you may say, "my face detector will first look for eyebrows, then the mouth, and only if both are present, it will say 'face' ".

"For simple machine learning models, like linear SVMs or linear regression, you can "look" inside the model and see what it is actually doing (well, sort of). The model in this case is just a weighted linear combination of the features and so a feature the model weights highly is probably important to the task, while a feature not weighted highly is probably irrelevant (this is a big oversimplification).

"However if your model is learnt, end-to-end, as well as highly non-linear, as in a neural network, you can't do this. You know the model is some non-linear combination of some neurons, each of which is some non-linear combination of some other neurons, but it is near impossible to say what each neuron is doing."

https://www.quora.com/Why-are-artificial-neural-nets-black-b...

>when thorough examinations from real brains just came out to the considerable detriment of said hypothesis.

I disagree that this work is detrimental to the idea of a grandmother cell. What this work shows is how a brain "understands" a face, i.e. how it extracts the meaningful features from a given face. But recognizing individuals is an orthogonal concern to "understanding" the face. This work shows how to decode/encode pixel intensity arrays into facial features. It doesn't show how the brain then maps a given set of facial features to individuals.

There are two things that ML/AI developers are going to have to deal with once the technologies become widespread in things like self-driving cars, hiring/firing decisions, and the criminal justice system:

"Why did it do that?"

and

"Make it stop doing that!"

The first time a self-driving car accident results in a court case, these things are going to come up. I very much doubt that people are going to be satisfied without clear explanations, and they shouldn't be. When these systems take on roles of increasing importance to society, some level of accountability is going to be necessary.

If I'm reading this correctly, it's old news. They're just tracing the activation of kernels. You can see examples in this wikipedia article: https://en.wikipedia.org/wiki/Kernel_(image_processing)

This one's cool too: http://scs.ryerson.ca/~aharley/vis/conv/

I like it. I've seen experiments that break out eigenvectors of a neural network, which is like being given a dictionary in a foreign language. It's precise, but you still have to figure out what each eigenvector means. This technique is like having a translating dictionary. It's less precise but it lets you reason about the network with a familiar visual vocabulary.

How are eigenvectors/values defined for nonlinear systems, and are their properties as useful as in the linear case?

I was thinking of https://en.wikipedia.org/wiki/Eigenface (I like slide 66 of https://www.slideshare.net/MostafaGMMostafa/neural-networks-... "Not magic: Can only capture linear variations".) But you can do things like Google's Deep Dream, where you pick the classification first and feed it backward to see what aspects of the image are important to a class. You can even do it for different layers to see what each layer is looking for. https://research.googleblog.com/2015/06/inceptionism-going-d...

Question to the community. Where can i follow research on this area understanding decision making and reasoning of neural networks?

Well, I think the research is more ad-hoc than being it's own field at this point.

I just scan papers that come up in the Reddit group[1]. I've seen:

"Chains of Reasoning over Entities, Relations, and Text using Recurrent Neural Networks" by Rajarshi Das, Arvind Neelakantan, David Belanger, Andrew McCallum

"Rationalizing Neural Predictions" by Tao Lei, Regina Barzilay and Tommi Jaakkola

"'Why Should I Trust You?' Explaining the Predictions of Any Classifier by Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

You might be able to chase down the works of these various authors to find more.

[1] https://www.reddit.com/r/MachineLearning/

I wonder if our understanding of neural networks will help us understand the human brain.

If THAT happens, then we can - after some further research - loose the last bits of our privacy - our mind...

Maybe one day. From my understanding we're not even close. We don't even have the computing power available to simulate a human brain. But yeah, scary applications. Minority Report anyone?

Probably not, considering the brain is not layered and iterative like (non recursive/recurrent) neural nets are. The similarities generally stop after the resemblance between a perceptron and a neuron.

Pretty but how does this provide insight?

When the computer is putting things into categories, you can see what aspects of the images are important to the decision.

Reply on news.ycombinator.com