Hacker News Reader

Comment by Borealid | Hacker News Reader

I think intuitively, intelligence is in some form about being able to predict the reaction that follows an action.

The extreme of intelligence is being able to create perfect statistical distributions for the effect of an intervention in the world. If you had that ability, you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X.

I don't present any support for or evidence of:

- LLMs gaining intelligence by whatever means

- Extreme intelligence being possible

- Extreme intelligence being feasible

But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.

View on HN

You seem to describe humanity and AI's ability to predict and manipulate humanity a bit like how a physicist describes the ability of an apparatus to affect a given dynamical system. So let me put it like this: (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|, and (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large (or that f_actual(X) saturates its codomain) when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."

That's a very sophisticated way to say "words can't convince people to do things".

I'm not sure that's a strong position to adopt in our world, but following your example I also won't try to persuade you otherwise.

That's not what I said, and it's the least sophisticated way I saw to phrase it to engage your argument in a precise way. The closest thing to your summary that I would accept is, "words can't convince people to do arbitrarily crazy things, in a way that you can predict."

> If you had [superintelligence], you could turn it on the question of "what would this human choose to do if I were to give them instruction X" for any X ... But I don't think I really need to provide support for the idea that something superintelligent would also be able to predict the actions of people under pressure in the same way someone of moderate intelligence can predict the actions of fluids under pressure.

So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

> (1) You cannot model the effect f_predict(X) of your probe X with arbitrarily low |f_actual(X) - f_predict(X)|

"You can't know what the impact of your action will be if it doesn't change much"

> (2) you cannot guarantee the impact |f_actual(X)| is arbitrarily large

"You can't necessarily change much"

> (or that f_actual(X) saturates its codomain)

"People can't do arbitrary things". I agree with this statement, but... people can build atomic bombs by themselves? See The Atomic Boy Scout. I don't think it's very relevant to the discussion that you can't make someone phase through a wall.

> when X in our case is restricted to inputs of the form, "tokens chosen in an LLM conversation."

"Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".

> "You can't know what the impact of your action will be if it doesn't change much"

Incorrect translation. No "if" clause need apply.

> "People can't do arbitrary things"

Also incorrect. f_actual(X) is a product of both "how much people can do" and "how much you can control/persuade them." I would model each of these with their own precise terms and relate them back to f_actual(X), and we could continue like this indefinitely, but from now on I'll just give you my reply in English: Not only does The Atomic Boy Scout not prove that a random person can bring about mass destruction (collecting random radiological material and contaminating the local area is not developing a nuke, nor is it even an effective dirty bomb, because it was neither very dirty nor a bomb; the safeguards were not lack of knowledge, either), but it still remains to be demonstrated that individuals will be radicalized by AI just by virtue of the AI being, like, super-duper smart.

> "Yes, I meant all the above specifically in the context of talking to a person, so I can't change my point of view later to be about something other than the persuasive power of words".

What domain of input do you propose, then? LLMs as rational agents have no agency but for what tokens they generate. What we hook them up to, and what they could plausibly do with what we hook them up to, has been left unsaid.

Denying the premise, which is that an intelligent being is one who can predict the outcome of actions with sufficient precision to influence the future, makes debate pointless.

I literally granted you that premise:

> even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

You don't need perfect prediction, you need good-enough influence on enough of the population. Which is something that adtech, social and trad media, and troll farms have perfected already.

We literally have people running around convinced the earth is flat. And a president who is both senile and delusional with access to the nuclear codes.

Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.

Or possibly inserted itself into critical elements of the tech industry. A digital adversary could do huge damage just by taking AWS offline.

That's just one possible attack vector among many others, most of which aren't well-defended. Especially if the AI itself is being used to harden their defences.

By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors?

The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.

> Especially if the AI itself is being used to harden their defences.

Your use of "is" rather than "isn't" only makes sense to me if the former interpretation is intended. To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented. (Unless you mean the bad actors will be the ones in control of all AI, i.e. OpenAI and Anthropic and friends?)

> A digital adversary could do huge damage just by taking AWS offline. That's just one possible attack vector among many others, most of which aren't well-defended.

I think you picked AWS because it's one of the few high-consequence digital systems a rogue ASI could plausibly take down on its own. Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.

Note: This is a very good argument to never remove the air, time, and meat gaps from those systems. If they do get removed, it will not be the genius of AI, but the stupidity of mankind that dooms us. Even Qwen could kill us at that point :)

>By "hostile ASI," are you speaking in terms of an AI that decides to go rogue independent of human instructions? Or of AI being used by bad actors? The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.

You seem to be moving the goalposts a little, but I don't see why "rogue AI" is less credible after the LLMs. It's obviously more. And we're just at the start of constantly running agent style harnesses.

>To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented

And yet, in many cases, those on the offense have a big advantage that any defense lacks. Especially if there's no concern for one's own life, which whether it's an AI or a human AI user removed from the field via his use of AI to delegate doesn't need to have (or they're just mad/fanatics and don't have in general).

>Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.

As if Stuxnet didn't exist?

> Hostile ASI wouldn't look like a rogue individual. It would look like a faction, group, or cult which gradually infiltrated and took over political power.

Or like a massive AI datacenter boom of questionable long-term profitability? Maybe the end goal isn’t profit but large amounts of compute?

>So can humans predict these things to a degree. But even having perfect prediction (which I contest with (1)) does not entail perfect control, just by continually computing the effect of, and refining, your X.

And why do you need perfect prediction and/or perfect control? "Good enough for the results you want" prediction and control sounds perfectly fine.

We have had cult leaders that had neither perfect prediction or control, and still convicted hundreds of thoysands of people to live as they say, and a decent chunk to even suicide on command!

Reply on news.ycombinator.com