Comment by Asooka | Hacker News Reader

Comment by Asooka | Hacker News Reader

Parent Full thread

But what do you expect to hear from the algorithm? As in, how "deep" should the explanation go, and what do you do if you don't like the explanation? How do you verify that the explanation is correct?

Edit: I should add that I worked on a similar problem. I can't reveal all the details, but it was about scoring pairs of objects to say how much one object is like the other. At first we used an off-the-shelf system that only spit out abstract values, so you can say "A is like B with a value of 5, but it's like C with a value of 5.17, so A is more like C than like B". The problem was those scores didn't correspond to any explainable metrics, so we switched to a custom system that could spit out a normalized value (0 to 1) and tell you exactly which qualities were alike and how the score was formed. People were NEVER happy with the values. Never. Everyone "felt" that these two should be more alike, or there should always be at least one object that has at least 0.8 likeness to another, etc. Plus, the values were now non-transitive, i.e. A can be 0.9 like B, but B is 0.2 like A. After about a year of tweaking we switched back to a non-explainable system.

Of course, the "non-explainable" system is perfectly explainable, it's a computer program and it isn't even deep learning, it's just long equations, the explanation is just "this long equation results in this value". The semantics of that equation though are very hard to impossible to understand. Plus, I don't think we can have a semantic explanation of a deep learning system precisely because we use deep learning when we can't semantically specify an algorithmic solution to the problem.

But what do you expect to hear from the algorithm? As in, how "deep" should the explanation go, and what do you do if you don't like the explanation? How do you verify that the explanation is correct?

It depends who you are explaining it to, of course, different audiences will have different standards, and will have or have access to, different kinds of experts to validate the explanation.

If the question is "why did your self-driving car hit that pedestrian?" for example, the standard would be pretty high. "It works most of the time but we don't know why it failed this time and we don't know when it might fail again" isn't going to fly. If the question is "we've just busted this trader for insider dealing, and you made the same trades at the same time, why?" then "the computer just told me to for no particular reason" isn't going to go down well either. You would need to show how data that you legitimately had access to reproducibly leads to exactly the same decision being made.

What do you do if you don't like the explanation? That's easy. Someone's going to jail. Don't let that be you :-)

> why did your self-driving car hit that pedestrian?

It didn't see him/her. Which is the extent of an explanation a human can give you. Try to picture wanting a person to explain why they didn't see a person coming from the side. We don't know what's happening inside our own heads.

That's my point - we're recreating an inherently non-explainable system in software, because we can't explain how it works. The algorithm is so complicated, we can't code it explicitly, so we let the machine figure out the semantic-free equations. All is not bad though, you could always dry-run the algorithm at home, i.e. ask your car "how visible am I with these clothes on", and just not walk outside with low-visibility and low-recognition clothes. Some countries already require pedestrians to wear reflective clothing in the evening to make themselves more visible. I suspect we'll have similar rules for self-driving vehicles: you must wear recognisable clothing when walking through traffic.

What I mean about "what if you don't like the explanation" is something else. Let's say you use a deep-learning software to determine sentences. The software spits out "5 years, because this is a black male". Do you accept that? It has been proven effective in many previous cases, can you accept that it has independently discovered a correlation between the person being black and the length of the sentence? Let's say you entirely remove the person's colour and sex from the dataset and re-train, then it will give you shorter sentences than you expect, because every judge previously had subconciously taken all these variables into account. Or it will discover a correlation between wearing e.g. adidas and alcoholism because Slavic people have a higher rate of alcoholism and wear adidas more often than others.

Which comes back to my core argument: If you know the acceptable metrics and algorithms, why are you using deep learning? If you don't know the acceptable algorithm, how can you verify the explanation?

It didn't see him/her. Which is the extent of an explanation a human can give you. Try to picture wanting a person to explain why they didn't see a person coming from the side. We don't know what's happening inside our own heads.

There is a world of difference between "this individual driver made a human error" and "software that can't reliably see people is driving a million cars now". The developers of the software would need to demonstrate they understood why the software failed in this situation and how they are going to fix it.

I suspect we'll have similar rules for self-driving vehicles: you must wear recognisable clothing when walking through traffic

Bear in mind that we can't even make cyclists now wear hi-vis or fit lights - making the entire population wear certain things for the benefit of job-destroying technology isn't going to get many votes!

If you know the acceptable metrics and algorithms, why are you using deep learning?

I disagree, because it's a matter of scale and usefulness. An algorithm that detects pictures of cats can answer the question of "why do you think that's a cat?" with "because it looks like a cat" and that's a perfectly reasonable explanation that a human might give[0]. But the stakes are raised when you start to deal with real people in the real world. "Why did you turn down that loan application" or "why did you add that name to the no-fly list" or "why did you reject that job candidate" need a bit more exposition.

That's a self-correcting problem because no-one sane will risk their business on software that "inexplicably" just happens to reject everyone in a protected class because there happens to be some vague correlation between members of that class and some undesirable activity or outcome.

[0] Except http://www.bbc.co.uk/news/technology-33347866 of course

> you must wear recognisable clothing when walking through traffic.

Which would make Driverless cars a non starter in the real world.

Dear God, I can't imagine the poor coder who has to dictate fashion so that car firms can ensure they don't hit people.

That addition reneges on the basic expectations people had of driver less cars in the first place, so now why would they have any motivation to let them happen?

Consider the following case.

Let's say we ban development of self-driving cars since the algos can't be explained. Then we will never know the benefit we might reap from adopting them. We know humans make mistake. But with self-driving cars, the error rates in future might be very low as compared to human. We as a human society will never experience this future because we had this silly idealism that all algos must explain themselves in human terms.

Let's say we ban development of self-driving cars since the algos can't be explained.

That's a false dilemma since it's by no means proven that the algos CAN'T be explained - just that we don't know how to extract the explanation from the model yet. It's entirely reasonable to suspend real-world use until the maths catches up.

But we would never reap the benefits in the meantime. That was my main point. And this becomes crucial if the saving of deaths from accident is substantial as compared to deaths that happen due to AI errors.

Reply on news.ycombinator.com