Hacker News Reader

Comment by martindbp | Hacker News Reader

Call it what you will, but from what I can see progress exploded when theoretical justification was thrown out the window. Maybe intelligence just is messy at its core? Science is all about induction, but what if there are no simple rules to break this down?

The only successful intelligence we know of came about through evolution, not science. The way I see it, researchers are now doing what evolution did, but in a more guided way and much more quickly. Whether you call that science or engineering or tinkering, it doesn't really matter.

View on HN

Actually most of the scientific progress in these tasks had already been made and all that was missing was the compute power. Just like the know-how and basic principles of how electric cars and have been around for more than a 100 years and what’s most been missing are cheap and powerful enough batteries.

So it depends on how you measure progress? Scientifically and advancement of knowledge, or success in applying such knowledge to business problems and use cases. That the deep learning community insists on publishing most of these breakthroughs as research is the basis of the critique being made in the OP.

This isn't correct.

No amount of compute power will let a RNN/LSTM outperform a Transformer.

No amount of compute power will let an AlexNet or Inception style network train as well as a ResNet.

> I think the fact that we're seeing so much progress in deep learning without any theoretical basis

I don't understand what you mean by this? Any neural network is first build in theory and then tried out practically by coding and testing. I think practitioner and scientist play equal role.

I think it was Nassim Taleb[1] that made a case for that engineering and tinkering drives science rather than the other way around, and that most of the knowledge gained comes from the practitioners rather than the scientists, who swoop in to claim credit after the fact (he uses the jet-engine as an example of this). That's not to say fundamental science is unimportant, just that it's getting too much of the credit for all the stuff that's been built.

There's a tremendous amount of know-how required to successfully build an airplane, a jet-engine, a processor, or to successfully train a large neural network. And while theoretical physics no doubt can explain it all, it's not sufficient to build something that works. I think the fact that we're seeing so much progress in deep learning without any theoretical basis is another example of this.

[1] Antifragile, Chapter: History Written By the Losers

> progress exploded

Did it? Deep Learning provided us with some mindblowing performance art (FaceApp, DeepFake, etc.), but as far as I know any actual business value that claims to be derived from 'AI' is fake news.

Business value is hardly the only or even main determinant of progress.

Of course, just pointing out that not everyone's point of view agrees with the glib pronunciation that 'progress exploded'.

In my little world attitudes have soured on 'AI'; 'AI' is posed to become the next 'blockchain'.

This is the natural course of the hype cycle. Like kids with a new toy, infatuation and then boredom. But the underlying technology in this case is absolutely world changing. I think we are only a few orders of magnitude away from systems that are genuinely intelligent. GPT-5 or 6 will have your jaw on the floor and GPT-10 will probably be smarter than all of us.

I think we are only a few orders of magnitude away from systems that are genuinely intelligent.

A few orders of magnitude is potentially very significant.

It's worth remembering that despite the hype and fake-clever systems playing chess and constructing plausible sentences, we still have nothing remotely approaching AGI at the moment, and no reason to think it is on the horizon.

But I agree that the AI tools we have have made significant progress in the line of powerful un-intelligence, and will have significant impact in automation.

"It's worth remembering that despite the hype and fake-clever systems playing chess and constructing plausible sentences". I completely disagree. I'm a connectionist and subscribe to the "Meaning is use" mantra from wittgenstein, when it comes to language. Language is the medium of thoughts, if GPT-x understands language, it is a general intelligence. If you read into the the output of GPT-3 you can see glimmers of genuine intelligence. An intelligent machine is not something you design, it is a machine that designs itself. All that is needed is to step out of the way of the machine. All you need is gradient descent and extraordinary processing power, there's no magic sauce left to find. Being clever and creating complex learning strategies just buys you more power at the expense of generality. Transformers and convolutional layers and the other tricks we have are probably now enough to get us within shooting distance of AGI. If you look at the progress of machine learning the best predictor of performance is processing power. I see no reason why it will not continue to scale. In the minds of many there is something very un-satisfying, unsettling even, in the idea progress without understanding. But the very definition of intelligence requires that you give up the responsibility of understanding to the machine. Now there is probably an overarching theory here that we don't understand - why is it that search within the space of algorithms described by a neural network is an effective strategy when solving the problems we tend to encounter in the real world? And that probably isn't a pure theory question, it's a question about physics - why do the emergent physics of the world permit easy solutions within the space of possible functions described by neural networks? Why are these solutions discoverable? No one has a handle on these questions yet, I suspect it's a very very deep question and unlikely to be solved within our lifetime. So everyone should "stop worrying and learn to love the neural network"

In other words, shut up and calculate? :)

That's an interesting POV, "if it looks like a duck... it doesn't matter that we don't understand why".

I agree with your point that (as per evolution) there is no reason why we can't just set the initial conditions and set it running and with luck/wise choices (tho the latter implies some understanding, which we don't have) an AGI will emerge.

However, I don't see any evidence of that despite the impressive power of GPT etc.

One reason why is that while I agree that "meaning is use", the use in the human case is in our interaction with the world and our inner thoughts.

The use in the GPT case is more of a Chinese Room style affair, isolated, unthinking and without motivation or interest.

During the typing of this comment I've considered a myriad of things, some related, some not - for instance, going to get a coffee in a minute, turning away to respond to a Whatsapp message. This motivated, conscious and subconscious context switching is just one example - let alone the myriad emotions, memories, feelings, inner life and experience that drive and inform it.

Intelligence as surface is only plausible in the philosophic zombie sense.

And we have nothing yet that comes near to mimicking even that general intelligence at surface level, just (massively impressive) powerful niche-explorers and domain entity generators like GPT.

AI is like the internet pre 2000's or electricity a century ago - barely been applied to the world. It's not time to get disappointed yet.

It's also like AI last century also in the lack of progress towards AGI, so plenty of time to already have been disappointed - certainly nothing to get excited about.

I agree with both those claims completely. Absurd hype, discussed many times here on HN.

Voice recognition, translation, OCR, image classification, etc. is done through neural networks mostly nowadays.

If https://en.m.wikipedia.org/wiki/Google_Translate does not seem like having business value to you, then I don't understand what you mean by "business value".

> If https://en.m.wikipedia.org/wiki/Google_Translate does not seem like having business value to you, then I don't understand what you mean by "business value".

There's no way Google Translate will ever be monetized. So yes, it's effectively an expensive art project for Google, not something that brings real business value.

Hotword detection ('ok google') is a practical application of AI, but that's really slim pickings considering how hyped 'AI' was. (Also, the costs of data collection for this feature are still way too high.)

I think we are really torturing language here. I don't want to get into specifics of what constitutes business value, how to value brand and customer satisfaction and so on. I can add more examples like https://deepmind.com/blog/article/deepmind-ai-reduces-google... that shows how neural networks helps in the pure bottom-line sense you care about.

Let me close my argument: neural network techniques developed in the last ten years are super useful here and now, they are used by billions of people every day and they do make their lives easier.

> I think we are really torturing language here.

Not really. My point is simple: as of 2020, any proposition for a business to 'invest in AI' is a money-losing proposition. (Just like 'blockchain'.)

Why are you so confident Google Translate is not monetized?

"We collect information to provide better services to all our users — from figuring out basic stuff like which language you speak, to more complex things like which ads you’ll find most useful, the people who matter most to you online, or which YouTube videos you might like..."

> Why are you so confident Google Translate is not monetized?

Obviously I don't have access to Google's accounting sheets, but there's no way in hell Google Translate pays for itself if you take all the costs of R&D into account.

Reason being that professionals who might pay big money for it want stuff other than just shiny 'AI' - they might want stuff like professional dictionaries and cited sources, etc.

And obviously people who translate random webpages for the lulz won't be paying much for this service.

Google's main income comes from its adverts business and that heavily relies on data collected from its users. Mail, translate, maps, and search are monetized. It's just not you paying for them.

If you all of a sudden start using Google Translate for Spanish and you think the phrases you are translating won't get used to serve you more targeted adverts for your upcoming vacation, you're being a bit naive.

Google Translate is being monetized. It's as easy a guess as yours to say that there's no way in hell Google Translate stays around if it didn't already pay for itself a long time ago. But let's stop guessing please.

It's already monetised: https://cloud.google.com/translate/pricing

Here's the pricing for translate:https://cloud.google.com/translate/pricing

I personally know companies spending tens of thousands per month on Google Translate.

I'm at my third company making money from neural networks, so I don't know where you get this idea.

2 have been around text and one on video.

Reply on news.ycombinator.com