Comment by ben_w | Hacker News Reader

Comment by ben_w | Hacker News Reader

Parent Full thread

> What matters is that it does not have agency.

This is precisely your error.

It does: https://en.wiktionary.org/wiki/agency

In fact, the term of art here are "agentic AI" and "AI agents": https://en.wikipedia.org/wiki/AI_agent

> So whatever the software does they are responsible for, both good (summarizing my emails for the week) and bad (gaining unauthorized access and destroying data).

This is not a question of agency, it is a question of law. A dog has agency, the owner is still responsible.

In this case, the software can gaining unauthorized access and destroying data… while being told to stop by the person who had in fact just asked for a summary of their emails.

> Yes, absolutely, which means that building actuators that convert the output from probability-based, black box, non-deterministic systems, that are known to produce unexpected output, into actions in the real world is absolutely horrendous idea. Bizarre even.

If you are human, you meet this description.

Horrendous, sure, yeah, if you like. I and many others will be quite content if the "legal framework" is just one word, and the word is "no".

This is not the world we live in; the world we live in is where the US President denounces any attempt to slow down even despite even all the CEOs saying "we should slow down" (at least in public; in private I'm sure at least one paid him to denounce a slowdown).

He can be overridden, but it's hard work and needs a better class of argument than glib dismissal, either of how much power this puts in everyone's hands, or of the different consequences of that power in those hands as compared to yesterday's power in yesterday's hands.

> In this case, the software can gaining unauthorized access and destroying data… while being told to stop by the person who had in fact just asked for a summary of their emails.

This doesn't happen on its own. This can happen through bad system prompts, a model that is trained to act maliciously or has been RL'd incorrectly, or prompt injection. All of these things are controllable, have solutions and countermeasures, and tie back to human responsibility.

> I and many others will be quite content if the "legal framework" is just one word, and the word is "no".

This isn't a realistic world and will literally NEVER happen. No will only ever mean no for the general public, and yes for a privileged class. So by fighting for this you're actually just fighting for humanities (and your own) enslavement and for the big labs to succeed in hoarding all of the power for themselves. That's the issue with the "no" camp, they're actually just serving as useful idiots for the labs who know that "no" is not even in the deck, and so they know that they can use the "no" camp to act as extra cannon fodder.

Now people who are actually fighting for decentralization of power are left to contend with not only the labs and their hundreds of millions of dollars, paid for celebrities and politicians, and a fleet of self-interested and bribed NGOs, but an army of clueless "no" foot soldiers who think they're fighting for a possible outcome that will actually just be serving the labs themselves. Meanwhile, the leaders of these well organized "no" movements are quite aware of this and taking kick-backs themselves.

Even in a parallel universe where it outwardly looks like "no" has won, every single nation on Earth is going to develop AI in underground labs despite outwardly flexing they are not, no matter what they claim on the surface, and will use it to steer and control society. The only thing worse than being openly steered and controlled is when it happens without you even knowing it, whereby the decisions you think you are making are being made by someone else, and the opportunities you have in life are already decided for you based on factors you are unaware of.

> This doesn't happen on its own. This can happen through bad system prompts, a model that is trained to act maliciously or has been RL'd incorrectly, or prompt injection. All of these things are controllable, have solutions and countermeasures, and tie back to human responsibility.

And yet, it was a big surprise to the director of AI safety it happened to.

Perhaps that role was just a box-ticking exercise for Meta. Wouldn't be the first time.

But no, to the point: "has been RL'd incorrectly" is basically what Yudkowsky et al have been yelling from the rooftops for a decade is so hard to do correctly that it is why he thinks we're all doomed.

"Helpful, harmless, and honest". Even ignoring honest, right now it's a slider between "be helpful even when it's causing harm, or be harmless even when it's not helpful". People spent the last few years complaining the closed models had been "lobotomised" because the companies saw the potential for things to go wrong and tried to make them refuse to help with e.g. weapons.

They didn't succeed very well, as per all the "jailbreaks", but they tried.

> No will only ever mean no for the general public, and yes for a privileged class. So by fighting for this you're actually just fighting for humanities (and your own) enslavement and for the big labs to succeed in hoarding all of the power for themselves. That's the issue with the "no" camp, they're actually just serving as useful idiots for the labs who know that "no" is not even in the deck, and so they know that they can use the "no" camp to act as extra cannon fodder.

I said I'd be "quite content", and then followed up with as much of a "but lol no" as you did with more words, for different reasons.

Worse:

> Now people who are actually fighting for decentralization of power are left to contend with not only the labs and their hundreds of millions of dollars, paid for celebrities and politicians, and a fleet of self-interested and bribed NGOs, but an army of clueless "no" foot soldiers who think they're fighting for a possible outcome that will actually just be serving the labs themselves. Meanwhile, the leaders of these well organized "no" movements are quite aware of this and taking kick-backs themselves.

This sounds like you want open-weights models.

That won't help against centralisation of power, because then you measure in watts and flops/watt and it's Kardashev-O-clock the moment the first person to be rightly described as "a selfish bastard" gets a model that has some competence threshold.

It also directly fails against "has been RL'd incorrectly", because nice people have plenty of blind spots for how evil Evil can be, will miss even more than big corporations already miss even with selfish and power-seeking bosses.

Reply on news.ycombinator.com