Today’s dominant approach to A.I. has not worked out | Hacker News Reader

Today’s dominant approach to A.I. has not worked out

nytimes.com

139 pointscohaagen137 comments

I feel like this article would be warranted if ML results had stalled and not achieved anything impressive in a few years. But we had AlphaZero recently, and Duplex is pretty impressive. There's no indication that cool new stuff isn't forthcoming in the near future. It's entirely possible that the current tradition will prove to not be up to the task of building an AGI by itself, and we'll need to invent new techniques. But in the absence of better ideas, continuing to iterate on ML and neural techniques seems like a good approach.

Yes. One of the things that drives me nuts about AGI critiques is that nobody is able to define incremental progress towards it, but they will assert confidently that today's increments won't get there. When they make any prescription at all, notes like OP tend to suggest techniques they think might reach farther, instead of proposing goals that might validate advances in those techniques.

I think that's the stagnant part of AI philosophy - a hazy dissatisfaction with task-oriented incrementalism that hasn't been able to offer an alternative objective function.

AI "progress" is nearly always presented in a way that confuses people, and usually includes some amount of fear mongering. I think its a problem between research and PR in general, but its exacerbated by weak theory in deep learning.

When Duplex is presented, the researchers do not communicate the domains where the research is reproducible, and where it is not. There is not theory that describes what domains can gain utility.

It seems like the obvious outcome of presenting research, without a theory that the audience can understand, is that the research will be misunderstood. Press narratives are required to latch onto this every time there are eyeballs on an AI demo: "Turing Test?", "Ethical Questions?", "Job Loss?", "How advanced is AI?", etc.

In terms of a "goal that might validate advance", I think a theory that helps others to understand the limits of the research. When other researchers, beneficiaries of the technology, and the press understand the real scope of the research it reduces the need for critique like OP.

Totally agreed with this function of critique! It's worth poking a hole in hype and fear around AI, I just don't think comparisons to AGI are a key part of that. A similar article might be: AGI is not the main focus of AI today, and that's ok.

In general, both hype and fear around AGI give people a false impression of preparing for AI impact. A truthful narrative about AI progress would focus attention on immediate issues like economic change (good and bad) and systematized bias.

Duplex is a demo right?

If AlphaZero (which doesn't impact users directly) and Duplex (which isn't released yet) are the best recent examples I can understand why there's negative press appearing.

Duplex's underlying text-to-speech technology research (WaveNet) has produced several papers and is now in public beta. It represents a huge advance in text-to-speech fidelity, using a remarkably straightforward algorithm.

https://arxiv.org/pdf/1609.03499.pdf

https://www.isca-speech.org/archive/Interspeech_2017/pdfs/14...

https://arxiv.org/pdf/1712.05884.pdf

https://cloud.google.com/text-to-speech/

The first paper is almost 2 years old, and text-to-speech seems to be a relatively small component of Duplex.

Uh, what? How are users relevant to the question of whether the tech is progressing? Humans at a game with such a high branching factor is a fantastic achievement.

Because A.I. has been sold as revolutionizing peoples lives. The second sentence of the article says Sundar Pichai, claimed in an interview that A.I. “is more profound than, I dunno, electricity or fire.”.

But if people don't see any impact in their lives, from a technology that they are told will change everything, then it's not surprising that they'd think little progress has been made.

The first quote that came to my mind when I read this was Frederick Jelinek's

"Every time I fire a linguist, the performance of the speech recognizer goes up".

Hi NYT writers, "language is infinitely complex" and "statistical approach to ML isn't working. Let's replace it with logic/rules based AI" don't belong in the same article.

At the same time, the shallow “understanding” of state of the art NLP methods are still not what we’re looking for. As we blow through records, we need to carefully state our shortcomings so that we can look for the next step.

It’s critical to ask how we’re supposed to get to something better.

Or does it not concern you that these methods work by modeling words by their context, not their content?

This is a topsy turvy argument, ignoring the abject failure of the old school AI approaches in the Chomsky and MIT tradition and the stunning success of ML statistical approaches. One wonders how little background research the NYT did before they chose these authors.

Stunning success in creating Hal 9000s or image classifiers? Because traditional AI was aiming for generalized intelligence and not just success in narrow domains. Minsky's critique of modern AI (or rather the entire history of the field so far) is the lack of progress on common sense.

So an ML model can learn to recognize cat pictures better than humans. Great, but what does it know about cats beyond being able to correctly pick out pictures with them in it?

Being able to recognize and learn from the environment can be seen as a the first step to learning common sense.

Imagine learning common sense without any way of getting or making sense of sensory input. Also being able to perform actions in the real world and seeing results. To be able to learn on it's own seems to be the only way to learn and develop.

I strongly dislike the phrase “common sense”. I have yet to encounter even one example of “common sense” which accurately represents the world.

“Things fall when you let go of them” «Except flying things, things already on the ground, and probably other examples too.» “But that’s not what I meant, it’s common sense that’s not what I meant.” «Then it’s a tautology, things that fall when you let go of them, fall when you let go of them. The real rule (or rather, good enough at our scale) is Newtonian mechanics.»

Or similar conversations about being in two places at once.

When it comes to cats, what does common sense tell you — What cats do? How cats respond to things? How they interact physically with their environment? That’s all statistically learnable.

>> What cats do? How cats respond to things? How they interact physically with their environment? That’s all statistically learnable.

Not at all. At least not in practice and certainly not for any significant subset of the things that "cats do" or the ways they "respond to things".

You can certainly collect some examples of all the behaviours you describe above, but there are so many of them you are never going to have enough to model a cat's behaviour with anything like significant accuracy.

DGANs learn to generate realistic faces. Do you think it's impossible, because there are so many different faces?

The generation is limited to features very close to the ones seen by the network in examples - i.e., to a limited domain.

Here is an explanation of the limited ability of deep networks to generalise beyond their dataset, by Francois Cholet (maintainer of Keras):

https://blog.keras.io/the-limitations-of-deep-learning.html

(Shrug) NNs are universal function approximators. A critique of existing approaches is important, but it will not last.

> never going to have enough

“Hey Siri show me cat videos”

“Here are some videos I found of ‘cat’ on the web:” «Link to YouTube page saying “About 79,900,000 results”»

The parent discusses:

>> What cats do? How cats respond to things? How they interact physically with their environment?

Not "how does a cat look on youtube".

You’re the second person who’s tried to correct my interpretation of my own posts this week. What’s that quote about losing grandparents? :)

Anyhow, my point was that there is plenty of data about the sort of cat behaviour which humans collectively find notable.

I see what you mean- you're saying that you can find 79mil cat videos on youtube, so you have 79 million examples of cat behaviours.

That might sound like a lot - but it's still not nearly enough to model a cat's behaviour. To convince yourself that this is the case, subtract 79 million from infinity. The number remaining is the number of cat behaviours that a model trained on 79 million youtube videos would never have seen and therefore not know how to deal with.

See, the point is not how much data you have- it's how much data you're missing. If the amount of data you have is a tiny part of the whole, then you can't model the whole very well.

It's already hard enough to train an image classifier to recognise still images (video frames) of cats. You're proposing to train some kind of model (it wouldn't be a classifier anymore) to recognise -and reason about- not only the likeness of a cat, but the relation of a cat with its environment; with arbitrary environments and arbitrary entities in those environments. And the cat is interacting with those arbitrary entities in the arbitrary environments in arbitrary ways.

Seriously, you're looking at an extravagantly large number that nothing we have right now can handle.

What is the quote about grandfathers? ~.^

The quote is “to lose one is unfortunate, to lose two looks like carelessness.” One person misunderstanding me I can ignore, two in the same way in a short window is definitely a sign I communicated poorly.

Why do you believe there is infinite cat behaviour? Why would they evolve that?

Even if they did, the point of learning is to reduce a probability distribution from “everything is equally likely, from this cat pawing at a toy to pushing a pen in exactly the right way to forge my signature, from hunting for a mouse to mugging an old lady for a voting card and using it to cast a fraudulent vote in her name for the Natural Law Party at the next election” so the probably distribution — your expectations — fits in a finite brain and matches all one has seen (70 million videos only need to be 36 seconds on average to be a lifetime of nothing but cats).

That being the case, all one really needs to do for a ”common sense” understanding of cats is the set of things human are not surprised by cats doing.

As I said before, I don’t like the phrase “common sense” because it’s such a bad model for reality. That being the case, it doesn’t matter what a cat would do when, say, elected governor of a small town — common sense, right or wrong, would say “eat, sleep, meow” or similar. Probably varies by person, given how many complain that “people just don’t have any common sense these days”.

Edit: why do you think it’s hard to train a classifier to recognise cats? Google did that the unnecessarily hard way six years ago, now we have GANs that imagine into existence cat pictures, as a student project to help apply for a PhD: https://ajolicoeur.wordpress.com/cats/

>> Why do you believe there is infinite cat behaviour? Why would they evolve that?

An infinity of behaviours is not a distinct ability that has evolved to fulfill some purpose. Rather, it's the result of the animal interacting with its environment. The number of possible such interactions is infinite - or, well, most likely infinite.

An infinite number of combinations can arise from very simple processes- for example, an automaton that generates strings in the aⁿbⁿ grammar (n a's followed by n b's) can go on for ever. There is no reason to assume that a complex mind in a complex environment will ever run out of combinations of mind-states and world-states. Accordingly, there is no reason to believe we will ever be able to collect examples of all those combinations, and represent them in computer memory.

Edit: I'm not talking about a cat being elected governor here. Just ordinary real-world behaviours, like all the ways a cat may chase a mouse, say. Try to observe a cat and systematise its behaviour and see how welll you can do. Then try to do it with a computer.

With a computer, it should be easier, right?

* * * *

I wouldn't say that the point of learning is to reduce infinity to something manageable. I think it's more like animal minds, like ours, have some kind of ability to pick out what is relevant to a learning task from the infinity of available experiences (incidentally, that's the subject of my PhD thesis :).

However, even as our minds are able to perform this one simple trick, we have no clue how we do it and can therefore not yet reproduce it with our machines. The result is the current state of the art in machine learning: data hungry algorithms that require loads and loads of computing power to reach top performance. This reliance on large datasets and compute limits progress: so far we've seen results only in situations were there is sufficient data and computing power and always in restricted domains (cat videos, vs cats in the wild). In problems where either there is not enough data, or the data is not sufficient because the domain is too large and too unconstrained, like natural language or modelling individual behaviour, progress has been much slower.

In short, modern machine learning substitutes quantity for quality, which has proven successful in the short term but looks to be self-limiting in the long run (even before the time when we're all dead). Eventually, we'll need to find an alternative or progress will stall.

* * * *

>> Edit: why do you think it’s hard to train a classifier to recognise cats?

Yep, that's a good example of what I'm talking about.

The model in the link was trained on 9304 examples and it shows- you can see the smudges and deformation in the high-res image (and the last one, sent by another person). I can't find the original dataset, but the results look very homogeneous, so they're basically just reproducing the training examples faithfully without generalising well- in other words, overfitting. Which makes sense: 9304 examples are maybe OK for a school project etc, but nowhere near enough for a real-world application.

Not that I can see a real-world application for generating faces of cats, but the point is that if you just want to train a small model to see how this sort of thing works, then you can certainly do it with a few examples; but if you want something useful, that approaches state-of-the-art performance then you need access to a lot more data and a lot more computing power.

I think you're underestimating how hard it is to make machine learning algorithms work well. It is worth reading announcements in the lay press and claims in scientific papers with a critical, even strongly skeptical attitude. Just because Google says that deep learning is the bees knees, don't just accept it as fact. Try to repeat their feats on your own. See how far you get.

I'm assuming you haven't otherwise you wouldn't be asking that question :)

This is getting too long and detailed to use my mobile to keep replying in as much detail as it deserves. :)

I get the impression that either (1) you have a very different definition of “common sense” to me, or (2) you are no longer talking about it. Does this seem like a fair representation? If so, can you explicitly describe what you mean by “common sense”?

As for reproducing results: limited experience of simple things only. Full time job has gone from software to full-time carer for parent with Alzheimer’s, so I don’t have time for anything more complex than e.g. {train scikit-learn to read from scratch, then read all the digits in Shakuntala Devi‘s number, then calculating the answer to her famous question} and timing it as faster than the human visual system takes to go from a number appearing to conscious awareness of that.

You know, fun toy examples for whiteboard interviews.

Mainly I’m keeping up to date with the “Two Minute Papers” YouTube channel. Hopefully I’ll be able to apply to start a PhD when my family can take over care duties for me…

Quick edit: I think your definition of intelligence is equivalent to mine. Please elaborate why you disagree?

Sorry for the comment size! I use HN from a PC always and I tend to forget that's probably not the most common use.

You're right, I'm not talking about common sense. It was another commenter who mentioned it. I interjected that it's very hard to collect enough examples of "What cats do? How cats respond to things? How they interact physically with their environment?" to build a good model of cat behaviour.

I'll be honest and say I have no idea what is "common sense" in the context of cat behaviour. Not to mention that it's very difficult to agree on a definition of "common sense". Despite that, I think you'll find there's general agreement that machine learning models don't have anything that could be recognised as "common sense". One reason for that is that it's extremely difficult to collect training examples of "common sense", exactly because it's so very hard to define it.

Apologies if that was too much of a sidetrack from what you wanted to discuss!

I actually don't have a definition of intelligence :) I'm working off an assumption that there is such a thing, that it's one process or one set of processes and that we may be able to reproduce it on computers, at some point in the future. But not in my life time.

The great advantage of doing a PhD is that you have plenty of time to read up and experiment to your heart's content. I hope it all goes well for you and you can soon start your studies.

I'm sorry to hear about yoru parent. You both have my sympathy. Hang in there.

Thanks! I think we’re basically in agreement then, as all of my responses were predicted on the incorrect belief that you were using common sense as an argument against AI.

I certainly agree that humans can accurately extrapolate — for example what a cat is likely to do next — with what seems like less data than any current machine learning system.

I suspect have my suspicions why, but to keep this short I’ll only say “catastrophic forgetting”, and separately that the normal approach in ML seems to be like teaching kids “by asking them random questions from the set of things we expect them to know at 18”, to almost-quote one of the podcasts I listen to.

One person's unmet dream isn't someone else's (or a field's) failure. Anyone who thinks their definition of AGI won't be met by current techniques is free to develop an alternative approach. The second it shows signs of being able to do something useful, it will attract funding and attention.

(Usually, the folks doing the OPs kind of heckling are posturing to get attention and funding before demonstrating something useful.)

What "abject failure" is that? GOFAI fell out of favour because its funding was cut as a result of political decisions by bureaucrats that did not understand the field, not because the field itself decided it had failed in its goals.

Here's a good read to give some background:

Avoiding another AI winter - James Hendler

https://web.archive.org/web/20120212012656/http://csdl2.comp...

And a little on the Lighthill report from wikipedia:

https://en.wikipedia.org/wiki/Lighthill_report

See the links to youtube videos of the televised debate (feature Jon McCarthy and Donald Michie and, er, another gentleman) in the external links section:

https://en.wikipedia.org/wiki/Lighthill_report#1973_BBC_%22C...

GOFAI - logic-based AI- worked just fine. Its branch of research was cut short before we could see where it would truly lead. Of course modern reserachers will not tell you that- it's their job to promote their ideas and your job to know the history of AI before 2012.

Old school AI had stunning successes of offloading and combining human situational knowledge into automated systems (under certain conditions). New school AI has stunning successes of association and prediction on data sets (under certain conditions).

Neither seem to point to a grand unified theory of human-level intelligence.

The symbolic AI tribe failed to reach what that the ML one never attempted. All fundamental differences aside they are hardly competing at all (except for eyeballs and grants). ML has made fantastic progress in recent years, but it could still be argued that the single most important discovery in ML has been the discovery of application domains with extremely little cost of being wrong.

No so much a discovery, but previous AI researchers defined what was important as what their tools were suited for.

But a good counter example is speech. They tried very very hard for many years, with quite limited success. As it turned out, decomposition into semantics via different layers isn't so successful.

http://oxygen.csail.mit.edu/Speech.html [Circa 2001]

As for AGI, there is no supporting basis for the contention that it can only be achieved using semantic approaches, or that ML isn't a productive path.

When the Google defenders brought up Apple's original iPhone demo that only worked if you followed the happy path and compared it to this staged demo, I found the comparison wanting.

The iPhone crashing when you did X was a simple debugging exercise. Making a chatbot that can understand the variety of human speech, translate it to text, understand it and respond intelligently is a much harder problem.

Not particularly. Speech to text is mostly solved at this point. When constrained to a small set of domains, so is intent recognition and canonicalization. That's certainly the hardest part, but we are still able to do it within specific domains.

Once you have a canonical request, response has been solved for a while. really I think it's more likely that you consider the problems where you understand how one would debug to be simpler ;)

Is it solved for various accents?

really I think it's more likely that you consider the problems where you understand how one would debug to be simpler ;)

Fair point. But I do have a little experience with natural language processing. My corpus was this at level 1.

https://www.newsinlevels.com

When you restrain things to a specific domain, yes. Essentially if you know that the conversation will be about restaurant booking, and you only hear

S_____r___zed th___ay, you can reconstruct sorry we are closed Thursday with a decent level of confidence. So short of being entirely gibberish, yes.

How can you constrain it to a specific domain if I just want to to dictate a text message to someone? That task is far from solved.

In the context of Google Duplex, I'm almost willing to give them the benefit of a doubt. Understanding text well enough to parse intent is relatively easy once you narrow the domain to making appointments. It doesn't take sophisticated AI just a lot of data and a standard rules engine. Even parroting back text answers in a limited domain is easy to do and make it sound human.

When I say "relatively easy", I don't mean "some guy in his basement". I mean a bunch of capable developers.

What I'm not convinced about is that speech to text is good enough once you take into consideration accents even within a limited domain. Especially when the person on the other end doesn't know they are speaking to a computer and are not going out of their way to speak clearly and simply.

I don't know much about Google Duplex, but recently I had to do the following: call all dentists in 20 mile radius, and make an appointment if they can:

1. treat a 3 year old kid

2. accept insurance xxx

3. see us within 5 days, preferably at 9am.

Out of all which meet all these criteria, pick the soonest available appointment, if multiple available on the same day, pick the closest one.

How should I explain all this to Google Duplex? How can I make sure it understood me? What exactly is it going to say to the appointment desk person? How can I verify it made the correct appointment? Would it need to make all the appointments first, then cancel all but one?

I would hope that Google would be smart enough not to include medical appointments as part of its domain. They shouldn't want to go anywhere near the liability of processing a call that falls under HIPAA and the liability of getting it wrong.

It could just ask the questions and send the user the short list to work from.

"we thought about coming Thursday but then you are closed so now we thought of Saturday".

The important thing is that you have timing information and can more or less accurately count phonemes/syllables even if you can't identify them. So the sentence you suggest couldn't fit in the space I suggested.

> Not particularly. Speech to text is mostly solved at this point

There's still no software capable to understand me properly in French right now, speech to text sill has a long way to go.

Is that just a question of expending the resources on that domain, though? Untold man-hours and compute cycles have been spent on making English speech-to-text mostly work. I would be surprised if even a fraction of that energy has been devoted to other languages.

Let's say we have a set of techniques that can do convincing text-to-speech in limited domains, but those techniques are too cumbersome to solve it in the general sense, at least with current equipment and data.

Then there's the ability of humans to process speech, which works in a much broader range of situations and therefore suggests that there is a better way to do that sort of thing than the techniques we have right now.

Does this help reconcile the two seemingly contradictory comments, above?

The issue with French and why it works so badly is the assumptions people made when designing the speech-to-text by starting from English. The core issue is that spoken French and written French are two completely and widely separate languages, with a much much greater difference than English. So the current approach to try to use books and mapping words to them isn't just working well.

What I mean is the current approach is pretty limited.

> The core issue is that spoken French and written French are two completely and widely separate languages

In what way?

Written french is largely codified by the french academy which is very conservative (so the written language did not evolve much in the past 100 years) whereas the spoken language evolved independently. It's a bit like slangs in English if you want but to a whole new level, the two parts of the language don't have much in common nowadays. Tenses, pronouns, grammar, sentence construction and words are different.

It’s hard for me to believe the they have diverged that much. I’d expect that the one would drastically influence the other. But then I don’t know French.

> Speech to text is mostly solved at this point.

Might be expecting too much from our market economy, but I wouldn't call it solved until it can be done locally, without engaging third party machines in the cloud.

(Ironically, voice recognition was working locally in the past, with quite decent accuracy. Then it got superseded with cloud solutions.)

> with quite decent accuracy

Citation needed. I've tried commercially available systems about 15 years ago, and the results were universally awful.

It's the limited domain that I can see making it feasible. I'm not convinced how well it will work with accents.

>> Making a chatbot that can understand the variety of human speech, translate it to text, understand it and respond intelligently

> Speech to text is mostly solved at this point.

You are talking about mapping sound patterns to words. The parent is talking about understanding what the words actually mean.

Those are very, very different things. We are nowhere near having a computer program that actually understands arbitrary language anywhere close to a human level.

Duplex is a clear example of how far we have to go. They struggle to build a program that can operate effectively in a single, incredibly constrained domain.

We are when you constrain the domain, which is what I said.

> The crux of the problem is that the field of artificial intelligence has not come to grips with the infinite complexity of language.

I stopped reading at that point. The field came to grips with the infinite complexity of language since before it was called "artificial intelligence."

And now that the field has come to grips with the infinite complexity of language, when should we expect the field to act on that understanding instead of attempting to bypass it?

https://www.theatlantic.com/technology/archive/2018/01/the-s...

Machine translation is getting pretty good, simply from crunching on enough text. The complexity of language looks finite. Translation between all the European languages works fairly well. Asian languages, not so much yet.

Strong AI still doesn't work. "Common sense" remains hard. Unstructured manipulation is still not very good. But legged locomotion is much better, as is vision processing.

Combining machine learning and geometry has promise. Look at how Waymo does automatic driving. (Not Uber or Tesla; they have no clue how to do it safely, as their crash record demonstrates.)

We're way ahead of where things were in the "AI winter", 1985-2005. This time the startups make money and do useful things. Progress will continue because there is revenue. AI used to be tiny - about 20-50 people at MIT, CMU, and Stanford. Now it's at least a thousand times bigger. Progress will be made by brute force.

(Me: MSCS, Stanford, 1985. I met most of the greats of classical logic-based AI. Trying to hammer the real world into predicate calculus just doesn't work. The expert systems guys were in denial big-time about this.)

Can you elaborate specially on Waymo's approach of combining machine learning and geometry, or point to a resource?

> We're way ahead of where things were in the "AI winter", 1985-2005.

That was the second AI winter. The first AI winter was in the early/mid 70s.

It is not inconceivable that a third AI winter will happen eventually.

I think that you've got your timing out there. Money started to flow straight after Gulf War 1 in 1992/3 because several folks had build components of the logic systems that were used for organising the logistics (Tate : HTN's; Winston schedulers - although I am hazy about this) What I am clear about is that there was money to be had for AI research after that.

The second AI winter is a quote from grandparent, not my own addition.

Sorry to drop in here and interrupt off topic, but you recently wrote a great comment that I missed the first time around, and now can't reply to, and don't know how else to get in touch (consider this "moving along"). It's off-topic but at least vaguely AI related. ;)

https://news.ycombinator.com/item?id=14983831

That is some fascinating stuff I've never heard before. I would love to discuss it more and ask you some questions! Maybe you could make a posting about the Turing Institute or Lighthill Debate? (There's a wikipedia page about Turing but it's kind of dry and lacking of scandal and palace intrigue.)

I worked at the Turing Institute in 1992, and it was an amazing place with many great people, including Arthur van Hoff who developed GoodNeWS/HyperNeWS/HyperLook! Unfortunately I never got the chance to meet Donald Michie (maybe he dropped by but I didn't recognize him), but I know from his reputation what a great guy he was. I did hear a funny story about him:

Donald Michie once overheard his secretary telling someone on the phone how to pronounce his name (in a Scottish accent): "It's Donald, as in Duck, and Michie, as in Mouse." He was so pissed he refused to speak to her for a month! ;)

For what it's worth, that's a great way to remember how his name is pronounced!

When I was there in '92 it was being (mis)run by some upper crust Tory gentleman who specialized in bailing out failing companies. He was so uptight he got pissed off I'd written the address as "North Hangover St." on the white board! Sheez. Instead putting him in charge, they should have just called Old Glory:

https://www.youtube.com/watch?v=KXnL7sdElno

I've been writing an article (still in draft) about the work I did at the Turing Institute (HyperLook):

https://medium.com/@donhopkins/hyperlook-nee-hypernews-nee-g...

Fun times! Anyway sorry to interrupt.

It may, but the amount of momentum and funding available right now dwarf anything that happened so far. As long as the next breakthrough happens before momentum and funding run out the wheel will keep turning.

Formalisations do break down in the limit, Wittgenstein showed everyone that, but the denial is a deep and powerful choice "he must transcend these propositions and then he will see the world alright."

There were engine room troubles for symbolic AI, committed choice meant that reasoners were unpredictable and impractical. We saw 100k improvement of compute change sub symbolic AI, and it has done the same for symbolic, and there have been some really smashing advances in the algorithms. Answer sets explode all solutions and then eliminate contradictions, we couldn't have dealt with them in the 90's (much less in 1985) as the idea of having 10G data structures in memory would have made people lol... but now so what?

Probabilistic formalisms and MCMC have changed what we can write down and reason about effectively as well.

I was very interested to read the story a few days ago about the relationship between compute resource and results in deep learning. https://blog.openai.com/ai-and-compute/? I was more interested to read the HN reaction! My team has had lots of success creating models using our cluster (in 2014) and then a cluster of GPU's which we bought because we could see that this was the way to go. We haven't gone and spent 100x more since then, and yet we now can see plotted out that the leading teams are doing this. The hardware has not improved at a rate of doubling every 3.4mths, it's hardly improved at all for 12 mths. What's happening is that people are spending money to make waves.

This is denial big time. Brute force is a wall, you can climb a few bricks, but the top is unobtainable. There have been some improvements in tree search and there have been some insights in terms of where we are wasting time with optimisations that aren't working but I've always thought that DL is brute force at heart and I think we are where we are with it now, and Alphazero is where we are going to be until someone really sees a new way.

>> I was very interested to read the story a few days ago about the relationship between compute resource and results in deep learning. https://blog.openai.com/ai-and-compute/?

Aye. My reaction to that graph was that this rate of progress can't be sustainable. If industry was throwing good money after bad to find out for how long you can keep ussing bubblesort given more and more compute, before having to develop a better sorting algorithm- that's what it would look like.

>> Answer sets explode all solutions and then eliminate contradictions, we couldn't have dealt with them in the 90's (much less in 1985) as the idea of having 10G data structures in memory would have made people lol... but now so what?

ASP is interesting. There are symbolic learning techniques that learn answer set programs from data, did you know that?

I didn't know that anyone had done that for answersets, could you share the reference? But on the other hand Stephen Muggleton has been developing systems to learn parts of logic programs for a long time, also I wrote a genetic programming system to learn prolog clauses but that was vs a hand coded objective function rather than vs. data.

To be honest, I'm not terribly familiar with the bibliography on learning AS programs, but you could look for work on ASPAL, ILASP and ILASP2 by Alessandra Russo, Mark Law and Krysia Broda. The following are relevant, but not exhaustive and probably not the first papers discussing this but you can follow the refs :)

Learning Through Hypothesis Refinement Using Answer Set Programming

https://link.springer.com/content/pdf/10.1007%2F978-3-662-44...

Iterative Learning of Answer Set Programs from Context Dependent Examples

https://arxiv.org/pdf/1608.01946v1.pdf

Learning Weak Constraints in Answer Set Programming

https://spiral.imperial.ac.uk/bitstream/10044/1/33615/8/Acce...

Finally, this is from Mark Law's portion in the Logic-based Machine Learning course at Imperial College London:

https://www.doc.ic.ac.uk/~ml1909/teaching/Non-monotonic%20Lo...

Full disclosure: I'm a Phd student of Stephen Muggleton's. If you're interested in where ILP is nowadays, our group is currently working on Meta-Interpretive Learning, a recent advance that learns complete logic programs, with recursion and predicate invention from relational data:

https://github.com/metagol/metagol

... and I'm very excited about it :)

Oh super - would you have any time to show it to me if I toddled over to Imperial at some point ?

Absolutely. See my profile for my email.

I tend to stay at home and go to Imperial for supervisor meetings only, once a week, but I'm sure we can arrange something :)

My apologieis but there is an ocean of misconceptions and a galaxy of misinformation, in that comment.

>> The complexity of language looks finite.

I have to ask what your defintion of "language" is because by formal language theory the only languages that are finite are, well, finite languages- which are simple than regular languages, which can be descirbed by a regular expression.

And I'm pretty sure that nobody has ever managed to write a regular expression that can describe all of human language.

>> Translation between all the European languages works fairly well. Asian languages, not so much yet.

In fact, translation between "all the European lagnuages" does not work at all well. For limited domains and for languages with lots of examples of translation, it works alright. Stray from that assumption and the results quickly descent into incomprehensible gibberish.

>> We're way ahead of where things were in the "AI winter", 1985-2005. This time the startups make money and do useful things.

Making money is a measure of progress of the industry- not of the science.

>> AI used to be tiny - about 20-50 people at MIT, CMU, and Stanford.

That was true back in the '50s, when the field began. In the '80s and '90s, at the hight of GOFAI and expert systems, there were several thousands of researchers working on AI.

For example, the 5th Generation Computer project was a massive effort by the Japanese Ministry of International Trade and Industry which encompassed pretty much all of that country's computer industry with huge loads of funding- not to mention the reaction in the West who panicked thinking that the Japanese were aobut to do to their computer industry what they had done to its car industry.

You might have heard of the failure of the project- but it took thousands of people a long time to fail. So, no, AI was not tiny, by any sense of the word, for any time after the first years of its birth.

>> (Me: MSCS, Stanford, 1985. I met most of the greats of classical logic-based AI. Trying to hammer the real world into predicate calculus just doesn't work. The expert systems guys were in denial big-time about this.)

Last week I met a guy with a background in theoretical physics, who works in high-performance computing. Am I now qualified to contribute an opinion about theoretical physics and high-performance computing?

If languages were infinitely complex, humans couldn’t do it either.

> Just as you can make infinitely many arithmetic equations by combining a few mathematical symbols and following a small set of rules, you can make infinitely many sentences by combining a modest set of words and a modest set of rules. A genuine, human-level A.I. will need to be able to cope with all of those possible sentences, not just a small fragment of them.

The issue is less about understanding language and much more about creating models of "the world" through observations, and then applying those models to perform tasks or answer questions.

Parsing language is actually pretty easy; "knowing" how it relates to a generalized model of "the world" is the hard part.

> No matter how much data you have and how many patterns you discern, your data will never match the creativity of human beings or the fluidity of the real world.

This is _firmly_ in the Opinion section of the NY Times...

What Google seemed to do with Duplex might seem like a babystep, but it doesn't take much imagination to recognize very real possibility for so-called "genuine, human-level A.I." to -- gradually -- emerge.

Uhh, it does. Being able to have a fun date night conversation has little in common with making an appointment for a hair cut. Duplex is a codified set of rules — essentially a set of blanks that need to get filled in and an elegant way of pulling together a bunch of components to do so. That’s nothing like free form conversion which requires world knowledge, perspective (gained from experience), personality, creativity, emotional connection, etc etc.

There’s nothing new going on in AI vs the machine learning “pragmatic downgrade” of the 90s, just the accuracy is higher.

Parsing the language at some level is required for translation beyond simple 1:1 word mapping. Understanding is really a separate task that sits outside of language and would be just as hard with a very limited language designed to be easy to decode.

Correct me if I'm wrong – are you saying that one doesn't need to understand something to be able to translate it? Because that'd be a strange argument to make, looking at the state of Google Translate.

> looking at the state of Google Translate

last time I checked Google Translate was a seq2seq based model with symbol level embeddings.

The way symbol embeddings are generated doesn't lead to any real understanding of the sentence/ word/ language. It only really leads to understanding what is normally around that symbol in different situations. You could also very easily argue that the seq2seq model doesn't provide understanding, it only learns to encode the general meaning of the sequence in a fixed compressed format.

It's possible to argue that being able to compress a sequence into a fixed length vector requires understanding but I would argue that understanding requires more than just being able to do lossy compression on a sequence. In my view entity modelling and related problems are much closer to achieving some level of understanding. They at least are able to use context to figure out what parts of the sequence have what meaning and the relationship between separate parts of the sequence.

I'd be interested to hear your view of how Google Translate has understanding.

I think the parent actually argues that GT doesn't have understanding.

I didn't mean to say GT has understanding, quite the opposite ;) I meant to use it as an example of how translating without understanding doesn't go too well in many cases.

I think I might have misunderstood your original comment though, as we seem to mostly be on the same side of the issue. That being said, could you expand on this?:

> the seq2seq model doesn't provide understanding, it only learns to encode the general meaning of the sequence

Maybe I'm nitpicking, but "encoding the general meaning" sounds a lot like a form of "understanding", and I wouldn't say seq2seq does any of that. (That's getting pretty philosophical though...)

Human translators need not be subject matter experts. So, it's really a question of depth of understanding.

Clearly deep understanding is useful, but just as clearly you can make a useful translation without it. So, the question is how superficial can 'understanding' be while still being useful. And at what point do you need to treat it as something other than understanding?

As far as I can tell any trained model can be useful, but that's a long way from saying any useful model means it's understanding.

The reason language is infinite is that it's structured recursively. It's the same reason mathematical expression are infinite, because you can replace a number with a bracketed expression. But computers can understand arbitrarily nested and large mathematical expressions given enough memory. So I think that it's misguided to argue that infinitude and recursiveness are the main obstacle to understanding natural language. The real issue most of the time is ambiguity and a lack of contextual knowledge, and in the general case, contextual knowledge requires general intelligence.

It was nice when nerdy news were only written in nerdy magazines read by a minority of people. Nowadays the spotlight is on tech but the coverage is not geeky, it's always opinionated, dystopian and as bad as politics. They ruin the fun before it even begins.

This article has some serious problems, as others have already noted. But one point I haven't noticed anyone making is this: the article falls into the trap expressed in the old saw "once it works, it isn't called AI anymore".

That is, AI researchers achieve a powerful result, and as soon as it's achieved, it's immediately (or nearly immediately) dismissed as being an interesting AI result, and the bar is moved forward yet again.

One can almost picture AI researchers as Sisyphus, pushing the rock to the top of the hill, only to have it roll back down on them, over and over and over and over and over and over...

if it's working well enough within the constraint, couldn't we just brute force the entirety of human language, thereby removing the constraint at some future point?

I feel like there would almost definitely be unexpected difficulties with that approach, possibly fundamental ones. Human language is crazy.

Why have it talk to people taking a hair appointment? It is vastly simpler to have autonomous agents on both sides talk to each other.

Why have autonomous agents talk to each other? It is vastly simpler to fill out an online form.

Because then you have to negotiate your schedule yourself.

I think having AI agents that can fill out online forms for you would actually be really really useful, and probably a much easier problem to solve.

You're right. I misunderstood the GP. Thanks.

It's simpler to have hundreds of thousands of small businesses, from all walks of life, sign up for a high tech appointment scheduling software?

Costwise it could be a simple matter of giving the software away and a cheap Chromebook. Creating the software is less complicated than what they are trying to do now and Google could easily justify the cost by the amount of data they could collect and getting a foothold in thousands of small businesses.

I feel like a large number of businesses already use software for scheduling but it’s a matter of having an API that can be exposed to customers that isn’t really supported. So it might be easier than you’re leading on.

A large number of medium to large businesses do. The overwhelming majority of small locally owned businesses don't. They still represent the overwhelming majority of sales.

Furthermore, big companies all implement their own software or purchase a variety of systems, some of which have API's, all have different API's and none have open API's.

A system that can speak, universally, to every Salon (or even say... 60%) is actually considerably less complex than the universal API you're calling "easier".

This will allow Google to confirm that their search rankings generate real revenue for businesses, so it makes things simpler for Google (and it is high-tech so that makes it cool! /s).

I swear every time this comes up it’s like people have never heard of open table or yelp, both of which handle online reservations for lots of businesses.

because google does not control both mobile and landline phones (at least not yet).

It solves the decentralized chicken-and-egg problem. You can't expect every appointment-place in the market to implement an appointment system that will work according to some common standard (which doesn't really exist yet), and the booking service isn't particularly useful until (and unless) a critical mass of service providers are compatible.

This approach, however, uses the "existing standard" (phone conversations in English or near-English) and can be eventually replaced by directly making an electronic booking for the places that will support it.

Google already offers Google Places to all small businesses. It would be trivial to them to offer an appointment system for free through the web.

Hmmm, maybe that's part of their longer-term plans!

Offering a booking system to businesses is nowhere near sufficient, to offer this service to customers without being able to fall back to phone calls requires a near-universal adoption and that is an entirely different issue. This system can enable electronic booking even to places who don't offer electronic booking, won't offer electronic booking and don't want to do anything whatsoever to offer it, which in many domains means pretty much all places.

You can't force every service provider to use a web-based appointment system, but you can make such appointments without their explicit cooperation if they offer to do it over the phone.

700,000,000 years ago to 300,000 years ago: all of nervous system life gets by on "pattern detection" and "curve fitting" (Judea Pearl's term for deep learning).

Homo neanderthalesis or thereabouts to modern day: language, reason, cause and effect.

> Today’s dominant approach to A.I. has not worked out.

But "curve fitting" biological neural networks needed 2500 times more architectural exploration and enhancement to get to rudimentary language capacity than rudimentary language architectures needed to achieve all of human knowledge. That suggests we're doing the right thing in growing and enhancing deep neural network architectures now. Evolution suggests the next step to language and reason on the right architecture is comparatively easy.

>Evolution suggests the next step to language and reason on the right architecture is comparatively easy.

<Something> suggesting the next step to language and reason is the right path and easy is the rhyming history of AI for sixty years.

"This time it's different."

> <Something> suggesting the next step to language and reason is the right path and easy is the rhyming history of AI for sixty years.

If evolution can be taken as a successful search through the space of what is possible, language and reason are on the path from curve fitting. But calling this "easy" is far from my point. If evolution is any sort of guide, deep neural network architectures are where the real work is needed before language understanding is possible (contra Pearl, Marcus).

Why not both? It is like fighting between engineers of digital electronics and analogue electronics. I believe the practical applications should use hybrid approach.

I believe the practical applications should use hybrid approach.

FWIW, I 100% agree with this. At least in the short-term. I think any "real" AGI will use elements of symbolic processing and sub-symbolic processing expressed as probabilistic / statistical pattern matching using something like ANN's.

The question I have is whether or not, in the end, all of what we call "symbolic processing" can eventually be expressed as operations at that ANN level (that is, can the "separate" domains actually be unified)? Given what we know (or think we know) about how the human brain works, I lean towards the answer being "yes", but I don't necessarily think we have to have everything reduced to ANN-level before we can achieve useful AI's. Arguably that last sentence was sill, because we already have "useful" AI, depending on your definitions of "useful" and/or "AI". :-)

AGI is overrated we already have humans for that, machines should focus on the types of problems that humans cannot already solve easily or cheaply.

Someone sent me this article on IM. My (unedited) response:

nothing worse than someone who both doesn't understand wtf he's talking about and is very critical

either are ok on their own

>> He‘s wrong then?

well if either author can provide an article from 10-20 years ago, saying "in 10-20 years, we'll invent a single machine that can reach almost human-level translation proficiency, drive a car in complex environments, beat the best human player at logical games including chess and Go, beat the best human player at language/knowledge based games like Jeopardy, hold a lengthy conversation and answer questions, be able to accurately describe and caption images and video, and outperform humans at trading the stock market and detecting fraud but I won't be impressed because all of those things are simple" then it might have some weight

"The dream of artificial intelligence was supposed to be grander than this — to help revolutionize medicine, say, or to produce trustworthy robot helpers for the home."

robots: we already have the vacuum cleaners. Actual flexible domestic work is coming soon, the moment the military gets bored and releases some knowledge and/or people have finished making money from helping amazon pack boxes and starts making money from normal people instead: https://www.youtube.com/watch?v=rVlhMGQgDkY

medicine: same machines already being used to detect cancer and heart conditions more accurately than humans, as well as folding proteins to create new medicines. Not sure what more he wants..

"If machine learning and big data can’t get us any further than a restaurant reservation" because you know, the current state of machine learning has been around for, what, 12 minutes now and we're still stuck on making restaurant reservations. Def time to call it quits and start over

". But in open-ended conversations about complex issues, such hedges will eventually get irritating, if not outright baffling." — weird that it's pretty difficult to get computers to understand a very ambiguous, illogical and inefficient form of communication. Luckily humans are so much smarter than computers that we can easily process 1-billion+ logical inferences a second and talk to them in an efficient and logical way.

>> Okay you‘ve convinced me

but i've only just started

> but I won't be impressed because all of those things are simple

When professionals in the field repeatedly say "X requires general intelligence" and yet time after time we solve them with shockingly simple techniques, the correct response is not to deride the AI for performing under expectations. It's to recognize that professionals have systematically overestimated the intrinsic difficulty of tasks, even really complex ones like photorealistic image synthesis, and propagate this belief through to everything else they say is hard.

I don't know what the next groundbreaking AI achievement is going to be, but I'd be happy to bet that the solution is going to be simpler than people expect. And while this doesn't necessarily generalize to everything, since there's a selection effect (though I suspect it's secondary), it does generalize to a lot of very important things.

I came to say roughly this but you’ve said it with more style.

Who is to say that in 15 years we won’t make difference-of-kind advancements in, say, natural language sythesis by virtue of a broad patchwork, piecewise sort of landscape made up of advancements in restaurant reservations, calendar items, customer service chatbots, etc.

Each solution region would be driven by incremental research and profitability constraints, and it’s fine to point at any one solution region and say, well that probably won’t generalize to something “AI complete” in natural language.

But as each bespoke arm of research gets more advanced, we could learn metaheuristics about selecting between them, or distilling common solution architectures.

People, even an informed academic in this article, still seem to appeal to some vague idea that “real” human level skill at a generic sythetic task like natural language has some mystical greater theoretical structure to it.

But time and again we see that the steady accumulation of a frankensteined patchwork of many approaches and many bespoke, situation-specific architectures can end up with uncanny, competitive results on tasks of import.

I’m not saying there’s anything about current fashionable methods that gives any guarantees, and of course there is always overblown hype, especially in corners of industry that try to use that hype for credential signalling and rent-seeking opportunities.

But it just seems short-sighted to come across as so certain that ugly patchworks of different siloed solutions could never be glued together or agglomerated to make difference-of-kind improvements. In fact, applied machine learning seems to a historical trail of exactly this type of patchwork.

Are there going to be cases when a whole new theoretical paradigm is needed to reach some performance goal? Yes, of course. And any sensible current practitioner would admit as much.

But this is a reason to throw the baby out with the bathwater?

The "simplicity" of tasks that mostly just require a lot of computing power was recognized over thirty years ago: https://en.wikipedia.org/wiki/Moravec%27s_paradox

> it is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility - 1988

Oh, you never ran into Hugo de Garis?

iirc (it was back in 1997 when I heard him talk), he was promising human level "artilects" about 2015, and super-human intelligence by 2025.

The problem that I have observed, is that the popular press always ignores the 100 experts that say, nope, not going to happen, for the one "visionary", who'll promise whatever they want to hear, as long as somebody else is paying for dinner.

If de Garis is (still) predicting super-human intelligence in under 10 years, then it sounds like he is impressed.

One correction: the military absolutely does not have any more advanced technology than the private sector. Very much the opposite, they've been totally incompetent at figuring out how to build software, or even buying software from the traditional contractors, which is the motivation for eric schmid'ts defense innovation stuff and Google's project maven. Many of the statements of the current DoD secretary flat out admit this stuff (plus if you've had the misfortunate working for a traditional DoD contractor it's also laughably obvious).

"Not worked out"? It seems like it's working pretty well to me. What a silly article.

Zombie Searle

>The crux of the problem is that the field of artificial intelligence has not come to grips with the infinite complexity of language

Or, put another way, the problem space reality presents is unimaginably large. So large that it took 4 billion years to achieve human level intelligence. That said humans have been working on the AI problem a pretty short period of time and we have made some pretty good strides at reproducing intelligence.

yeah that's the point. you don't write down a million rules. you write down a program that can find the rules.

That's sort of the crux of the startup I work at - www.gamalon.com

> the problem space reality presents is unimaginably large

For our purposes as a heavily time-limited species, it's infinite. Across the globe we'll probably create hundreds of thousands of narrow AI programs over the coming decades and that won't touch a meaningful fraction of what could be done. AI writing AI is the only way to jump that a lot further. I'm skeptical of general AI over the next 20-30 years or so, however I do expect we'll commonly see AI writing narrow AI programs in the next 10-15 years.

Most of my communication is online now, so its a format much easier for machines to parse.

If the problem space was as large as they suggest it is, human brains wouldn’t be capable of handling it either.

I think the authors are saying that the number of mental models (possible worlds) that language can suggest is unmanageably infinite, unless the listening brain is capable of rapidly eliminating all the infeasible / implausible worlds before forming a viable model of what the speaker is thinking, before formulating a response. To do that, the listener needs to bring to bear a wealth of world knowledge and common sense, all of which is lacking in the current models of AI created by deep learning.

As such, unless AI can somehow build useful models of the world and combine them with basic horse sense, it will always be just a one trick pony.

> it took 4 billion years to achieve human level intelligence

Using random mutations over the space of DNA, happening at random intervals.

You know, it's kind of funny. We have no idea if the current path is trending toward a local optima, or if we're heading toward a global optima.

We are completely aware of the issues of finding a local optima in our AI (especially ML) algorithms, but little thought has been given if our AI evolution has been trending toward one or not. We're pouring vast research reserves along a few pathways, and hoping that our hill climbing leads to a global maxima (or at least a local one sufficiently high as to be 'human level intelligence')

Are we?

It seems quite the opposite, that the vast research reserves of AI, now heavily funded and employed by industry, are applied to pathways on hills that credibly seem high enough. Current deep learning approaches are sufficient for systems that work well enough to make profitable products, and that's it, there's no claim, expectation or necessity for that to be even a local optimum, much less a global one.

> but little thought has been given if our AI evolution has been trending toward one or not.

Wait, you think people aren't thinking about that all the time? They absolutely are. And there's very little obvious ways to determine if this is a local or global optima. Resources are limited and everyone wants to follow the current best results, because that's necessary for business success.

One would expect a competitive advantage, then, in funding research to try something else. Exclusively following the current best results just means equivalency, and provides no competitive edge.

Evolution wasn't trying very hard to create intelligence; it's really not that useful for survival relative to how expensive it is biologically, to the extent that some people think it's more an artifact of human mate selection and signaling (like a peacock's tail) than a proper adaptive trait.

Without a strong survival gradient pushing a population towards higher intelligence, it's actually surprising that evolution stumbled on it at all, which speaks to how easy it must be to find in solution space, not how hard.

I would be very surprised if a human-guided search can't crack the problem relatively soon - we're only just now starting to hit the threshold of computing power necessary to do it, so it's not too shocking that we haven't fully succeeded yet.

You read The Mating Mind too, huh? :)

Yes, and thank you, I couldn't remember the name of the book to reference it!

I think it makes a somewhat strong case; to me, the big problem to explain regarding the evolution of intelligence is the fact that it only helps survival a little bit, not a lot. And it evolved in very little time, which usually requires huge selection pressure. Sexual selection offers a potential explanation there.

Earnest Davis from NYU right? Next bit of news we’ll hear is “Yann Lencun starts fist fight with fellow professor”

This group does not know what they are talking about. Mark my words here. It's about advanced distance calculations and engineered feature attributes.

No it’s not. A complete definition of intellect (let alone consciousness) has never been done. When it is then progress will be made.

Attempts at it have https://arxiv.org/abs/1709.08568

Why would we need that (and why do you think it is even possible)? We didn't have a definition of spacetime until we had one. Our current definitions if intellect and consciousness may be completely off, like the concept of absolute time.

Reply on news.ycombinator.com