Hacker News Reader

Comment by AndrewKemendo | Hacker News Reader

No, you're completely on the right track.

People usually judge AI systems based on superhuman performance criteria with almost no human baseline.

For example, both Google Translate and Facebook's translation system could reasonably be considered superhuman in performance because the singular systems can translate into dozens of languages immediately more accurately and better than any single human could. Unfortunately people compare these to a collection of the best translators in the world.

So you're exactly on track, that humans are heavily prejudiced against even simple mistakes that computers make, yet let consistent continuous mistakes slide for humans.

View on HN

>Unfortunately people compare these to a collection of the best translators in the world.

I don't think that's really true. They're comparing them to the performance of a person who is a native speaker of both of the languages in question. That seems like a pretty reasonable comparison point, since it's basically the ceiling for performance on a translation task (leaving aside literary aspects of translation). If you know what the sentence in the source language means, and if you know the target language, then you can successfully translate. Translation isn't some kind of super difficult task that only specially trained translators can do well. Any human who speaks two languages fluently (which is not rare, looking at the world as a whole) can translate between them well.

> Any human who speaks two languages fluently (which is not rare, looking at the world as a whole) can translate between them well.

No, sadly, that's not the case. A bilingual person may create a passable translation, in that they can keep the main gist and that all the important words in the original text will appear in the translated text in some form, but that does not automatically make it a "well-translated" text. They are frequently contorted, using syntaxes that hardly appear in any natural sentences, and riddled with useless or wrong pronouns.

A good translation requires more skill than just knowing the two languages.

Right, but I don't think many people expect machine translation to do better than that. And I don't think the ability to do good translations is quite as rare among bilinguals as you're making out. Even kids tend to be pretty good at it according to some research: https://web.stanford.edu/~hakuta/www/research/publications/(...

> They are frequently contorted, using syntaxes that hardly appear in any natural sentences, and riddled with useless or wrong pronouns.

I find it hard to believe that this is the case if the translator is a native speaker of the target language. I mean, I might do a bad job of translating a Spanish text to English (because my Spanish sucks), but my translation isn't going to be bad English, it's just going to be inaccurate.

Consider yourself lucky, then. You are speaking English, with its thousands (if not millions) of competent translators translating everything remotely interesting from other languages. The bar for "good translation" is kept reasonably high for English.

As a Korean speaker, if I walk into a bookstore and pick up any book translated from English, and read a page or two, chances are that I will find at least one sentence where I can see what the original English expression must have been, because the translator chose a wrong Korean word which sticks out like a sore thumb. Like, using a word for "(economic/technological) development" to describe "advanced cancer". Or translating "it may seem excessive [to the reader] ..." into "we can consider it as excessive ..."

And yes, these translators don't think twice about making sentences that no native speaker would be caught speaking. Some even defends the practice by saying they are faithful to the European syntax of the original text! Gah.

> Like, using a word for "(economic/technological) development" to describe "advanced cancer"

That sounds like a mistake caused by the translator having a (relatively) poor knowledge of English. A bilingual English/Korean speaker wouldn't make that mistake. I mean, I don't know your linguistic background, but you clearly know enough English and Korean to know that that's a bad translation, and you presumably wouldn't have made the same mistake if you'd been translating the book.

>Some even defends the practice by saying they are faithful to the European syntax of the original text!

I think there's always a tension between making the translation faithful to the original text and making it idiomatic. That's partly a matter of taste, especially in literature.

> A bilingual English/Korean speaker wouldn't make that mistake.

Well, "bilingual" is not black and white. I think you have a point here, but considering that people who are paid to translate can't get these stuff right, the argument veers into the territory of "no true bilingual person".

Anyway, my pet theory is that it is surprisingly hard to translate from language A to B, even when you are reasonably good at both A and B. Our brain is wired to spontaneously generate sentences: given a situation, it effortlessly generates a sentence that perfectly matches it. Unfortunately, it is not trained at all for "Given this sentence in language A, re-create the same situation in your mind and generate a sentence in language B that conveys the same meaning." In a sense, it is like acting. Everybody can laugh on their own: to convincingly portray someone else laughing is quite another matter.

Perhaps they're already using mechanical translation and then only correcting sentences that are basically ungrammatical, not just weird.

People paid to do something are rarely the best at it.

They are however consistent since their pay check depends on it.

>Well, "bilingual" is not black and white.

Not entirely, but it is definitely possible for someone to be a native speaker of two languages, and they wouldn't make those kinds of mistakes if they were.

>They're comparing them to the performance of a person who is a native speaker of both of the languages in question.

Which is synonymous with the best translators in the world. Those people are relatively few and far between honestly - I've traveled a lot and I'd argue that native bi-lingual people are quite rare.

Depends on which part of the world you're in. Have you been to the USA? English/Spanish bilingualism is pretty common there. And there are lots of places where it's completely unremarkable for children to grow up speaking two languages.

https://www.researchgate.net/post/What_is_the_percentage_of_...

This is well said, but one reason this double-standard is rational is that current AI systems are far worse at recovery-from-error than humans are. A great example of this is Alexa: if a workflow requires more than one statement to complete, and Alexa fails to understand what you say, you are at the mercy of brittle conversation logic (not AI) to recover. More often than not you have to start over or worse yet execute an incorrect action, then do something new to cancel. In contrasts, even humans who can barely understand each other can build understanding gradually because of the context and knowledge they share.

Our best AIs are superhuman only at tightly scoped tasks, and our prejudice encodes the utility we get from the flexibility and resilience of general intelligence.

> our prejudice encodes the utility we get from the flexibility and resilience of general intelligence

I don't think any particular human is so general in intelligence. We can do stuff related to day to day survival (walk, talk, eat, etc) and then we have one or a few skills to earn a living. Someone is good at programming, another at sales, and so on - nobody is best at all tasks.

We're lousy at general tasks. General intelligence includes tasks we can't even imagine yet.

For thousands of years the whole of humanity survived with a mythical / naive understanding of the world. We can't even understand the world in one lifetime. Similarly, we don't even understand our bodies well enough, even with today's technology. I think human intelligence remained the same, what evolved was the culture, which is a different beast. During the evolution of computers, CPUs remain basically the same (just faster, they are all Turing machines) - what evolved was the software, culminating in current day AI/ML.

What you're talking about is better explained by prejudice against computers based on past experience, but we're bad at predicting the evolution of computing and our prejudices are lagging.

I might use this in future critical discussions of AI. “It’s not really intelligent.” Yeah, well, neither am I. On a more serious note, it seems obvious to me that technology is incremental, and we are where we are. Given 20 more years of peacetime, we’ll be further along. When VGA 320x200x256 arrived it was dubbed photorealistic. I wonder what HN would have had to say about that.

Yes, that's almost exactly how I start most of these conversations: "Do you know how BAD we are at most things?"

Venkatesh Rao had a great piece on how mediocre we really are recently:

https://www.ribbonfarm.com/2018/04/24/survival-of-the-medioc...

Being able to do many things at a level below what trained humans can isn't what any reasonable person would call superhuman performance. If machine translation could perform at the level of human translators at even one pair of languages (like English-Mandarin), that would be impressive. That would be the standard people apply. But they very cleary can't.

It's a problem with who you're comparing against.

Generally people think superhuman = better than the best humans. I understand this and it's an obvious choice, but it assumes that humans are measured on a objective scale of quality for a task, which is rarely the case. Being on the front line of deploying ML systems, I think it's the wrong way to measure it.

I think Superhuman should be considered relative to the competence level of average person who has the average amount of training on the task. This is because from the "business decision" level, if I am evaluating between hiring a human with a few months or a year of training and a tensorflow docker container that is reliably good/bad, then I am going to pick the container every time.

That's what is relevant today - and the container will get better.

> Generally people think superhuman = better than the best humans.

Because that's what it means.

Also, I shudder to think of the prose coming out of a company that decides to "hire" an AI instead of a translator...

Well not explicitly or in any measurable terms [1]. The term 'Superhuman' lacks technical depth in the sense of measurement. So for the purposes of measuring systems we build vs human capability, it's a pretty terrible measure.

[1] http://www.dictionary.com/browse/superhuman

Reply on news.ycombinator.com