Fireside Chat with Ilya Sutskever and Jensen Huang | Hacker News Reader

Fireside Chat with Ilya Sutskever and Jensen Huang

youtube.com

57 pointsbelter15 comments

Maybe something should be added to this post tile to indicate it is 6 months old for context?

Here is a more recent podcast episode (No Priors) with Ilya Sutskever: https://youtube.com/watch?v=Ft0gTO2K85A At 38:34, he says they’ll have some very exciting research to share soon.

Interesting analogy: https://youtu.be/GI4Tpi48DlA?t=1041

It’s not that interesting. He argued against the nonexistent straw man that no system that can predict the next word could be doing reasoning, which is self-evidently false, as you are perfectly able to predict words yourself.

The actual question is whether focusing on improving word prediction is the way to achieve reasoning.

If a system could reliably predict the last word of a detective novel, it would arguably be doing reasoning, sure. If it could reliably predict the last word of “the simple one word solution to global warming is”, it would be really amazing. That isn’t an argument that such a system can exist or that you know how to get there.

GPT4 isn’t just doing word prediction though. It has image input in GPT4V, and image output via Dalle. OpenAI are well on the way down the path of multimodality.

It would be fair enough to say that GPT3 doesn’t know what a dog is, it only knows about the word “dog”, but GPT4 has a conception of a dog that is both visual and literary, and as they add more modes the depth of understanding will increase.

Dalle is a whole separate model that isn't multimodal; GPT4 just write prompts. If you play with it you can actually see dalle usually doesn't listen to most of what it writes.

GPT4V seems to be truly multimodal but is also a different model from GPT4; the writing style is different and it's maybe less intelligent.

https://www.adept.ai/blog/fuyu-8b

Doesn't seem like multimodal inference is as complex as you're making it out to be. It all hinges on what you claim "understanding" to be; are trained weights understanding?

I think you are replying to an argument he was not making. :-)

Really? “The better a neural network can predict the next word in text, the better it understands it. […] People say that deep learning won’t lead to reasoning, but in order to predict that next word, […] who was the murderer, that requires some reasoning, a fair amount of reasoning […]”

Predicting the next word in that situation might require understanding. It doesn’t follow that trying to make it predict the next word better will result in understanding.

Yes. And at which point do people accept that this is the level at which plenty of humans operate? But the analogy fails because especially in mystery novels there is space for a lot of explicit logic reasoning: build a chart of which evidence exists, of what it means about each person, or what that means about the evidence, etc. Current models do not (yet) do that. Most mystery novels are NOT about predicting the next word. If anything they are anti-predicting the next word (because if that plot has already been done, that's a lot less valuable novel.)

But then also very few humans do such analytic decisions or charts. That's not how most humans live their lives. It does not make sense to place the bar for "Yep this is AGI" at "mystery writer" (or "super scientist"). Seems to me, mostly humans just predict the next word. Not even very well.

> And at which point do people accept that this is the level at which plenty of humans operate?

I think you underestimate humans. One thing that humans (and many animals) are that LLMs aren’t, is goal-orientated.

We are capable of recognizing something we want and planning and executing a course of action to achieve it - this includes what we say; what we say is determined by our goals/objectives.

LLMs do nothing of the sort. All their output is based on their training data - i.e. the “choices” aren’t made by the LLM but by the generator of the original training data.

Most things i say aren't really goal oriented. I would put it at 10%

If you are hungry you go find food and eat it. If you are bored you go find something fun to do. Most of our actions, if not all, are to satisfy one need or another. Satisfying that need is the goal.

Some humans - perhaps many here - are reasoning-obsessive-compulsive.

But most human it feels, and myself much of the time, are on auto-pilot. When I'm driving back home from an event I don't need to reason about anything. I can have a conversation with my friend next to me. -- and I will often even forget that I'm giving that friend a ride back and miss their exit - because I'm too much on auto-pilot. I am just guessing the next word (in my driving). It's a hyper-alert kind of guessing the next word but I'm not reasoning until something really unexpected happens. And even then I may not have the time.

I argue that even when I have to take the train and check the schedule, I do that on "guess-the-next-word" kind of auto-pilot. Yes I look up a database (train schedule) but I'm not reinventing the text. I've done it enough times, I'm looking up the database while guessing the next word. Totally within sight for LLMs.

You mention "capable". Yeah many humans are capable of reasoning through their day to day life. But many are absolutely not. If you ask them to enumerate the steps needed to take the bus, they will fail. Not because they can't take the bus but because they never think of it explicitely. Never "reason" about it.

You are still goal-oriented. If you are hungry, you go look for food then consume said food if you find any. If you are bored, you look for entertainment - e.g. calling one of your friends (who isn’t a complete bore). It doesn’t have to involve complex reasoning.

Reply on news.ycombinator.com