Hacker News Reader

If you think about the way that humans learn, we get formal education that the computer doesn't get. We are getting corrected for our mistakes by other people. This work did not originally do that. The newer versions of it get corrected by people but that doesn't mean that it couldn't be corrected by more instances of itself that operate over different datasets via a consensus algorithm. The algorithm will also probably get flack that it is not unsupervised when human learning is actually partially supervised.

One could argue that humans have a similar problem of errors propagating. Areas that we feel strongly about can bias us against learning in fields such as religion, politics, and, of course, programming language design.

I think that it is possible that humans would make some similarly poor responses in areas that we don't talk about often.

I agree absolutely. I believe that future AI's will be educated in much the same way we educate children. Perhaps less education will be needed - possibly we could start them off at a higher age bracket for example. But ultimately I'm certain the first "strong" AI's will be educated/supervised to some degree.

My point was more that perhaps we're a bit too focused on the wrong metric for success. The current criteria set is {positives, negatives, false positives, false negatives}, and we try to optimise for high/low degrees of one or the other in order to determine whether a particular approach is successful or not.

What is then overlooked, is that perhaps we don't need to have a near-perfect positive rate, but instead achieve an acceptably-incorrect false positive or false negative rate. Where the answer may be wrong, but it's not too far wrong. Much like a human might pin a country like India in the wrong place on the map, but wouldn't ever put it in the middle of the Indian ocean.

In summation: Perhaps the key for computers to appear intelligent, is not to be perfectly correct, but to be not too disastrously incorrect.

Perhaps you can extend the F1-score[1] to be an F1,w-score that weights errors based on some measure of distance from correctness.

[1]http://en.wikipedia.org/wiki/F1_score

Sounds like a good idea. Finding that distance heuristic will be a challenge though!

I agree. In fact, errors like thinking an internet cookie is a baked good sounds EXACTLY like the type or mistake a small child would make.

That makes me remember that as a small kid there were certain spots in my environment beyond which, I was conviced, infinite wilderness would adjoin or the world would just end (for example behind forests or hedges). Growing older I was often sobered when I discrovered there was just bog-standard urban landscape.

Yet, the machine seems to come from an entirely differernt direction. It has much more facts accumulated than a child and can articulate and process them to absolute precision.

I remember visiting my childhood elementary school and seeing that the "forest" behind the building was just a scraggly patch of trees with a chain-link fence on the other side. It seemed so much bigger back then...

So finding a way to mitigate computer mistakes, to make them milder, is every bit as important as eliminating them entirely - a potentially impossible task.

Seems like they should run multiple instances against different but overlapping corpuses and see what they agree on. For all we know, any single person would forget what a door knob is if they had all the information in the world in their brain.

But maybe it already works that way internally, I don't know.

> they're just far more likely to make wildly incorrect mistakes than understandably incorrect ones

Have you spoken to some members of the general public and asked what they think? People believe all sorts of wacky things. We are just a lot more tolerant of the mistakes people make. http://www.buzzfeed.com/mjs538/things-americans-believe-in

13% error rate in a self-driving car, hilarity ensues...

Exactly. A human would usually make "acceptably incorrect" decisions, such as not indicating, or speeding. But these mistakes generally don't end in disaster. Only occasionally making a really bad mistake ending in an accident. So out of those 13%, it might only end up with <0.1% overall as accidents, the rest being recoverable.

Whereas the uncontrolled randomness of computer mistakes could be horrendous. Hence why I'm saying an interesting avenue to look at in AI, might be to achieve acceptably incorrect answers (acceptable negatives), rather than just aiming for high true-positive and low false-positive rates.

Leave room for a margin of error, same as humans do.

Well reading through their site it seems it gets some categories of facts right nearly all the time, and some categories it does poorly at. This should be expected because some things are hard to understand even for a human, especially if you don't already know a lot about the subject. However it does seem to assign a confidence estimate to every fact it learns. So it does get some facts wrong, but at least it knows what probability there is the fact is wrong and doesn't assume it is 100% true.