Comment by lukev | Hacker News Reader

Comment by lukev | Hacker News Reader

This is a good discussion of concepts. However, a lot of the folks worried about superintelligence miss an important division within the concept of "intelligence." Specifically with respect to the current generation of LLMs.

Namely, there's a huge difference between the concept of intelligence and an intelligent agent.

You could make the argument that current LLMs have some level of intelligence. But (without a lot of supporting infrastructure) they are not agents, they have no built in mechanism for having any kind of intent or even short-term memory. Because they fundamentally just generate a single token at once, they don't even exist "within time" the way we normally would think of an intelligent entity as doing... they're more like a mathematical function that you can invoke on demand. Perhaps to get an "intelligent" result, but conflating that kind of intelligence with an intelligent entity or actor seems like a mistake.

To use the analogy in the OP, it would be like calling a single hydraulic piston "strong". Sure, but unless you attach it to something it's not the same kind of strong as a bodybuilder is.

OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT. So we should expect a "superintelligent agent" to appear pretty much the next day after a "superintelligence" is created.

Perhaps, I don't discount the possibility that an LLM could be used as a primitive as part of a larger more complex system that could rightfully be called intelligent.

But a script running a LLM in an OODA loop is a lot easier to monitor and control than some inscrutable "ghost in the parameters" that a lot of AI alarmists seem worried about. You can actually see a transcript of its "thoughts", for one!

We already have self-replicating scripts that can prove very hard to detect and destroy, even without the benefit of LLM-powered fuzzing and social engineering.

First of all, we're assuming that agents are (at least initially) designed intentionally, not as harmful viruses.

Second, running a LLM inference (nevermind training!) involves lots and lots of system resources. Much more difficult to hide than a tiny backdoor.

OK, but the hard part was the "intelligent" part, right? If you have the "intelligent" thing, and you want to get an "intelligent agent", all you have to do at most is wrap it in a shell script that runs the equivalent of an OODA loop, e.g. AutoGPT.

No, not really.

Neither AutoGPT or any similar actually work. And the only justification for expecting them to work the future is same naively extrapolation more of one kind of quality will automatically give a thing more of a correlated quality.

The humming bird building airplanes: "If we continue to increase the capacity of these flying machines, eventually they will have 'super-bird abilities' and at that point they will naturally steal all of nectar..."

AutoGPT doesn't work because GPT-4 is too stupid to make good plans, not because it's missing the secret sauce of "agentiness." If some future software was really great at observing, planning, and reasoning, then an AutoGPT-like wrapper would make it good at autonomously doing things, too.

Optimality is the tiger, and agents are its teeth: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...

As someone who has worked/is working on building LLM driven agents, this allegory basically reads to me like:

1. I ask a LLM for paperclips. 2. ~Magic happens~. 3. Humanity is doomed.

Even if a LLM was good enough to drive a system like this (current ones are not, by a long ways), there's a million reasons it would not and could not play out this way.

I think (from the outside looking in) many lesswrongers are slowly coming around from the "paperclipper" scenario to something more gradual, in response to how the tech seems to be developing currently. More along the lines of "we hand over bigger and bigger parts of running the world to AI systems and this gradually leads to less and less human involvement (until such time as the intelligent systems that run the world see humans as an unnecessary interference that can be done away with)". Something like that at least.

Some pointers here, maybe: https://astralcodexten.substack.com/p/davidson-on-takeoff-sp...

I mean, to me this sounds like "some christians are coming around to the fact that cosmology is mostly correct about the shape of the universe, but they want us to say that God created the big bang since you can't explain it yet."

Which is to say, right or wrong, it's a narrative in search of confirmation more than it's an actually rational understanding of what is real.

IMHO you're doing it a disservice by calling it a "narrative" and comparing to christian cosmology. On the face of it I think ideas like "past some threshold AI can become dangerous to humans" or "AI can become more intelligent through a positive feedback loop", even if speculative, have some plausibility and are worth investigating.

You are basically chiding people for changing their mind in the face of new information, while still staying committed to some core ideas based on argumentation which is not challenged by this new information. Sure, this roughly looks like what happened to religions as science developed, but it looks just as much like how any scientific or philosophical field of inquiry makes progress.

Given that the "lesswrongers" are just a bunch of pontificating blowhards, why should we take them seriously on anything?

The basic idea of the essay, I think, is that "an agent" is a thing in idea-space that a non-agent intelligence (oracle) has access to and can describe to a user. "describe" in this context means "write code to represent" Then the user is turning what was an oracular AI into an agentic AI. The idea here, is that if you think that you're safe because you haven't made an agent, just an oracle, that's a false sense of security. Oracles are just one user choice away from becoming an agent. If one thinks agents are safe for some reason, that's a whole separate discussion I think.

Reply on news.ycombinator.com