Hacker News Reader

Comment by Rotten194 | Hacker News Reader

I get the author's feelings about why we shouldn't tell this story, but I still disagree. It's a pithy, funny example of GIGO in machine learning. People could read conclusions about the abilities of neural networks from the story, but they're wrong to do so -- it's a PEBKAC error, not a technology one. "Truthy" cautionary tales are a near-universal feature of human cultures -- why shouldn't machine learning have some?

View on HN

It's a plausible story. Fine to present as a parable, but we should stop presenting it as true unless we can find a reliable source for it.

Until today, I believed it was true. It was told to me as an undergrad, by a professor who believed it himself.

I first heard it in an AI class as well. When I relate the story I usually give it as an anecdote of apocryphal origin. But I do love the story, because it brings into focus several issues: what do you actually want your classifier to learn? what does your training set actually teach? how do you know you've learned the right thing?

Many times it seems like people go into these things hoping that the machine learning part will figure out things for them and relieve themselves of the problem of thinking hard about the problem. It doesn't. It only moves your problem over a bit and increases the difficulty.

In fact this problem pops up even in pedagogy where the lessons people are taught actually train them to do the wrong thing (for example pilots responding to aircraft attitude upsets).

The parable's lesson is a simplistic one, basically: "stop and think about what you're doing". But like other simple lessons about crying wolf or stitching in-time, it bears repeating.

> It only moves your problem over a bit and increases the difficulty.

Well, that's not quite true. In robot sensing several things have recently moved from the nigh-on-impossible column to the holy-shit-that-actually-works-pretty-well column, thanks to ML.

But I agree with the rest of it.

> I suggest that dataset bias is real but exaggerated by the tank story, giving a misleading indication of risks from deep learning and that it would be better to not repeat it but focus on established risks like AI systems optimizing for wrong utility functions.

He's not arguing against having cautionary tales, he's arguing that we should base them on actual problems instead of imaginary ones.

But ensuring correct datasets is a problem that people new to machine learning have to be made aware of. It's easy for people to see that NNs are good with 'noisy' data and incorrectly assume they can throw any data at it and get good results. And I think the quote is a false dilemma -- its not like we can't have multiple different stories for different problems. Make up / find a "truthy" story to spread for misoptimized utility functions if one doesn't exist, but there's no need to kill this useful story in the process.

If a failure mode has never been reported in the wild, why is it so important to tell a juicy story about it, at the expense of attention to empirically observed failure modes?

Google's image tech confusing black people and gorillas?

Sure, it wouldn't do any harm to move to relevant and well-sourced examples, or construct such demonstrations specifically for the class. This HN discussion alone has gathered several instances of classifiers overfitting or fitting to features that do not generalize outside the training set as well as it first may appear, e.g. the curious case of leopard sofa [1] linked by andreasvc [2] demonstrates more or less same problem as the tank parable (leopard detector performs quite well but in reality is not detecting the exact thing we'd expect).

However, also consider that the tank parable has circulated in textbooks and undergraduate introductory lectures for several decades. The main lesson I learned from the story that after training a model, one should validate it to see if it does generalize to detecting tanks both during night and daytime. Is it really a surprise that it might be difficult find egregiously naive mistakes?

[1] http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht...

[2] https://news.ycombinator.com/item?id=15486441

Reply on news.ycombinator.com