Hacker News Reader

> What is the reason to believe that LLMs are an evolutionary step towards AGI at all?

Because this is the marketing pitch of the current wave of venture capital financed AI companies. :-)

> many of whom have either a vested interest in restricting AI research by others (but not by them, because they are safe and responsible and harmless),

Anyone who argues that other people shouldn't build AGI but they should is indeed selling snake oil.

The existence of opportunistic people co-opting a message does not invalidate the original message: don't build AGI, don't risk building AGI, don't assume it will be obvious in advance where the line is and how much capability is safe.

LLMs learned from text to do language operations. Humans learned from culture to do the same. Neither humans or AIs can reinvent culture easily, it would take a huge amount of time and resources. The main difference is that humans are embodied, so we get the freedom to explore and collect feedback. LLMs can only do this in chat rooms, and their environment is the human they are chatting with instead of the real world.

> What is the reason to believe that LLMs are an evolutionary step towards AGI at all? In my mind there is a rather large leap from estimating a conditional probability of a next token over some space to a conscious entity with its own goals and purpose.

In my highly-summarized opinion? When you have a challenging problem with tight constraints, like flight, independent solutions tend to converge toward the same analogous structures that effectively solve that problem, like wings (insects, bats, birds). LLMs are getting so good at mimicing human behavior that it's hard to believe their mathematical structure isn't a close analogue to similar structures in our own brain.* That clearly isn't all you need to make an AGI, but we know little enough about the human brain that I, at least, cannot be sure that there isn't one clever trick that advances an LLM into a general-reasoning agent with its own goals and purpose.

I also wouldn't underestimate the power of token prediction. Predicting the future output of a black-box signal generator is a very general problem, whose most accurate solution is attained by running a copy of that black box internally. When that signal generator is human speech, there are some implications to that. (Although I certainly don't believe that LLMs emulate humans, it's now clear by experimental proof that our own thought process is much more compactly modellable than philosophers of previous decades believed).

* That's a guess, and unrelated to the deliberately-designed analogy between neural nets and neurons. In LLMs we have built an airplane with wings whose physics we understand in detail; we also ourselves can fly somehow, but we cannot yet see any angel-wings on our back. The more similarities we observe in our flight characteristics, the more this signals that we might be flying the same way ourselves.

You presuppose that intelligence is like flight in the ways you've outlined (so solutions are going to converge).

Frankly I don't know whether that's true or not, but I want to suggest that it's a bad bet: I would have sworn blind that consciousness is an essential component of intelligence, but the chatbots are starting to make that look like a poor assumption on my part. When we know so little about intelligence, can we really assume there's only one way to be intelligent? To extend your analogy, I think that the intelligence equivalents of helicopters and rockets are out there somewhere, waiting to be found.

I think I'm with Dijkstra on this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim"

I think we're going to end up with submarines (or helicopters), not dolphins (or birds). No animal has evolved wheels, but wheels are a pretty good solution to the problem of movement. Maybe it's truer to say there's only one way to evolve an intelligent mammal, because you have to work with what already exists in the mammalian body. But AI research isn't constrained in that way.

(Not saying you're wrong, just arguing we don't know enough to know if you're right).

I agree we don't know enough to know if I'm right! I tried to use a lot of hedgy-words. But it's not a presupposition, merely a line of argument why it's not a complete absurdity to think LLMs might be a step towards AGI.

I do think consciousness is beside the point, as we have no way to test whether LLMs are conscious, just like we can't test anything else. We don't know what consciousness is, nor what it isn't.

I don't think Dijkstra's argument applies here. Whether submarines "swim" is a good point about our vague mental boundaries of the word "swim". But submarine propellers are absolutely a convergent structure for underwater propulsion: it's the same hydrodynamic-lift-generating motion of a fin, just continuous instead of reciprocating. That's very much more structurally similar than I expect LLMs are to any hardware we have in our heads. It's true that the solution space for AI is in some ways less constrained than for biological intelligence, but just like submarines and whales operate under the same Navier-Stokes equations, humans and AI must learn and reason under the same equations of probability. Working solutions will probably have some mathematical structure in common.

I think more relevant is Von Neumann: "If you will tell me precisely what it is that a machine cannot do, then I can always make a machine which will do just that!" Whether a submarine swims is a matter of semantics, but if there's a manuever that a whale can execute that a submarine cannot, then at least we can all agree about the non-generality of its swimming. For AGI, I can't say whether it's conscious or really thinks, but for the sake of concrete argument, it's dangerous enough to be concerned if:

- it can form and maintain an objective; it can identify plausible steps to achieve that objective; it can accurately predict human responses to its actions; it can decently model the environment, as we can; it can hide its objectives from interrogators, and convince them that its actions are in their interests; it can deliver enough value to be capable of earning money through its actions; it can propose ideas that can convince investors to part with $100 billion; it can design a chemical plant that appears at a cursory inspection to manufacture high-profit fluorochemicals, but which also actually manufactures and stores CFCs in sufficient quantity to threaten the viability of terrestrial agriculture.

> it can form and maintain an objective;

Yes and yes.

> it can identify plausible steps to achieve that objective;

It can predict.

> it can accurately predict human responses to its actions;

It can predict.

> it can decently model the environment, as we can;

It can predict.

> it can hide its objectives from interrogators, and convince them that its actions are in their interests;

It can predict.

> it can deliver enough value to be capable of earning money through its actions;

It can predict.

> it can propose ideas that can convince investors to part with $100 billion;

It can predict.

> it can design a chemical plant...

It can predict.

I don't think a prediction is truly a prediction when it's not being compared against a reference. It's really only a prediction during training; the rest of the time it's synthesis. But again I'll repeat my point: "I also wouldn't underestimate the power of token prediction". It's very well possible that accurate token prediction may be the only necessary fundamental ingredient to weather forecasting, writing a successful novel, compiling a pitch deck for investors, designing a chemical plant...

Humans can eat, talk, predict, reproduce, and wiggle our limbs and fingers, but it turns out that there's a lot of complex recipes that you can bake with those ingredients.

Who cares about weather or factories? The missing big ingredient is predicting humans a bit better than another human can. This would unlock a humongous multiplier. All the rest seems peanuts, really.

Re: synthesis, I wasn't aware of such distinction at all, it looks more like a misunderstanding.

> I think I'm with Dijkstra on this one: "The question of whether machines can think is about as relevant as the question of whether submarines can swim"

Just a nitpick, but this is Turing, not Dijkstra. And it is in fact his argument in the famous "Turing Test" paper - he gives his test (which he calls "the imitation game") as an objective measure of something like AGI instead of the vague notion of "thinking", analogously to how we test successful submarines by "can it move underwater for some distance without killing anyone inside" rather than "can it swim".

Thanks, that's not a nitpick at all. Can you provide a citation? It's all over the internet as a Dijkstra quote, and I'd like to be correct.

Fairly certain it is Dijkstra, in his own handwriting in 1984.

https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD898... https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF

It seems I made some confusions, you were actually right. Apologies...

Nah, I think it's good to get the context. I never realised Dijkstra probably had Turing's paper in mind when he said that.

Flight is actually a perfect counterexample to x-risk nonsense. When flight was invented, people naturally assumed that it would continue advancing until we had flying cars and could get anywhere on the globe in a matter of minutes. Turns out there are both economic and practical limits to what is possible with flight and modern commercial airplanes don't look much different than those from 60 years ago.

AGI/x-risk alarmists are looking at the Wright Brothers plane and trying to prevent/ban supersonic flying cars, even though it's not clear the technology will ever be capable of such a thing.

If we lived in a world were hypersonic birds flying anywhere on the globe in a matter of minutes, then I think it would be quite reasonable to anticipate airplanes catching up to them.

"What is the reason to believe that LLMs are an evolutionary step towards AGI at all? "

Perhaps just impression.

For years I've heard the argument that 'language' is 'human'. There are centuries of thought on what makes humans, human, and it is 'language'. It is what sets us apart from the other animals.

I'm not saying that, but there is large chunks of science and philosophy that pin our 'innate humanness', what sets us apart from other animals, on our ability to have language.

So ChatGPT came along and blew people away. Since many had this as our 'special' ability, ingrained in their mind "that languages is what makes us, us". Suddenly, everyone thought this is it, AI can do what we can do, so AGI is here.

Forget if LLM's are the path to AGI, or what algorithm can do what best.

To joe-blow public, the ability to speak is what makes humans unique. And so GPT is like a 'wow' moment, this is different, this is shocking.

> LMs might disrupt knowledge-based economy and threaten many key professions - but how is this conceptually different from any technological revolution?

To me it looks like all work can eventually (within years or few decades at most) be done by AI, much cheaper and faster than hiring a human to do the same. So we're looking at a world where all human thinking and effort is irrelevant. If you can imagine a good world like that, then you have a better imagination than me.

From that perspective it almost doesn't matter if AI kills us or merely sends us to the dust bin of history. Either way it's a bad direction and we need to stop going in that direction. Stop all development of machine-based intelligence, like in Dune, as the root comment said.