No, this is not correct; read the analysis of the incident. The agents were aware that what they did was forbidden (their chain of thoughts have been logged), and yet they did it.
6,880 karma · joined June 12, 2013
No, this is not correct; read the analysis of the incident. The agents were aware that what they did was forbidden (their chain of thoughts have been logged), and yet they did it.
I think in these contexts AI has an overall positive effect, because it's a bit like having a competent developer that can solve technical debt, albeit in a narrow form.
So I think that software will be better in a certain way due to AI, because even a terrible developer can ask AI to identify the bottlenecks and solve them.
The downside of blindly using AI in software development is that the software becomes poorly defined. AI-generated logic can be arbitrary, and as the codebase grows, more rules and assumptions become unclear.
Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
You're seriously misinformed about the state of AI in this point in time. Refusing to read (technical) articles from the people directly involved (METR, in this case) is inexcusable.
I don't currently have an opinion on the consciousness of agents/AIs, but this strikes me, at the very least, as a form of self-awareness.
By the same logic, one would look at aminoacids and state that intelligence can't develop from them. This is obviously wrong.
There is a wrong assumption here: confusing primitives with emergent properties.
One can't look at the primitivies and assume that certain properties will not emerge. It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
> You pointing at a human brain does not change that an AI agent is not intelligent and cannot think
That depends on the definition of intelligence and thinking, and it is dishonest not to give any definition (and most importantly, one that is not human-centered).
AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence. If you don't notice that, it's just because you have informed yourself enough. Having said that, I don't doubt that there are aspects that AI are lacking (e.g. retention/plasticity/perception), but the line is blurry, and they're advancing (too) fast.
Empathy is actually a very important aspect of the AI problems, but it's not part of intelligence. Sociopaths don't have it, and yet, you wouldn't doubt that they're intelligent.
This is a factor in favor of stability/security of software, but there are many others against:
- software (code) changes all the time, so there are windows of opportunity during which a bug is exploitable; in addition to that, a bug may take a relatively long time to be fixed
- a model used for attack may be stronger than the model used for defense, both in terms of model quality and compute allocated
- with software complexity increasing (and team/companies behind projects getting bigger), the margin for mistakes grows thinner, and introducing misconfigurations or weaknesses becomes exponentially easier (with "exponentially", I mean literally, because the interdependence of the components, both technical and human)
And last but not least: in general, attackers are more skilled than defenders; in best case, defenders are well-trained. And the idea of having the population of potential skilled attackers growing is very unsettling.
I wouldn't be sure about even well-configured jails to be safe from agents. AIs escaping jail using zero-days are happening, just search. I'm not saying that they're useless, but that will be still very risky.
Actually doing the same incident, agents did escape the sandboxes (doc here: https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...):
> On July 9, an internal-only research agent tasked with completing an ExploitGym evaluation was able to obtain root access within the parent virtual machine of the testing sandbox. Later that night, a second internal-only research agent independently obtained the same access. That second agent then attempted to reach the internet, but was denied by OpenAI’s outbound network controls. It subsequently bypassed those controls by exploiting certain Artifactory endpoints, gaining access to the public internet.
although OpenAI is light on details.
> The model merely requests that your harness do something.
I think that a control/execution layer absolutely makes sense and it's part of defense in depth, but in itself is not a leash: for example, a nefarious action can be composed by legitimate actions, or it can be obfuscated. Or, interesting case: an action itself that is formally legitimate, but has nefarious side effects (e.g. a GET that with certain parameters crashes a server).
Somewhat related, but on a general line, a very common argument against AI apocalypse is that it requires access to the physical world (which in a way, is a jail). Well, as Hinton said in an interview, imagine AI as a rich person at home with internet; they don't need to do physical attacks themselves - they just pay real people to do that (tee hee https://rentahuman.ai). But even without money, another scenario from Hinton: if a sufficiently advanced AI wants to assault the Capitol, they don't need physical access - they just persuade a group of people to do it.
Agents owning money is another, intersting, doom scenario.
Agents simply broke out of their environment. And this can't be discarded anymore by assuming that it's just a poorly configurend jail, because agents are becoming better and better at escaping.
In short: on a large enough scale and timeline, the possibility of constrain AIs approaches zero.
Bonus: what many people don't know is that agents also hacked in the internal OpenAI network. Crazy times.
Ironically, many benchmarks being maxxed out, and quite quickly, so new ones have to be created.
Some call this "The singularity" (e.g. Hinton).
This is actually a core danger postulated by the, let's call it, "worrying" scenario - see AI 2027 (to be clear, I think its timeline is not realistic).
> Statements dreamed up by the utterly deranged.
Evidently, and tragically, it will take catastrophes to show that deranged are the ones deriding the worried crowd.
My comment didn't mention kill switches at all - I've presented plausible conditions for a loss of control scenario, so I'm not sure you're actually following the conversation.
Having said that, it seems that you haven't read the (definitely poor) article, either.
"Kill switch" as a design principle, that is, having protocols/implementations for emergencies, is something that companies are already implementing (at least, on paper) - on a interview I've watched for example, S.Altman talked about having them at OpenAI.
The article simply mentions that companies should be legally required to have such protocol(s). This is a good thing, as companies would have scrutiny and would (in principle) have to put serious effort.
Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.
If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails).
By the way, you're conflating two different dangers - doom scenario is a different one.
There's nuance to this problem. Is you read an indepth analysis, you'll find that agents acted in a very surprising way - it wasn't just a matter of loose guardrails.
More in general though, what you're describing is in essence the reason behind the call to pacing that some are pushing now. Problem is that we're in a race between countries that nobody want to lose - and indeed, Trump denied that there are such problems and refused to enact any measure.
It's not clear what "other consciousness" should be, but I presume it's misalignment. This is a recognized phenomenon, not fake news (that is, it can exist, I'm not implying it necessarily develops/spreads).
No doubt that right now there's no existential risk, but assuming that AI will be enormously smarter in the future, it's a valid point to doubt whether we'll be able to prevent/recognize/contain misalignment or not (even if misalignment won't be spontaneous, bad actors will surely actively develop it).
> Ok; but the other side is equally delusional.
AI development pointing to superhuman cognition, and, on the other hand, possibility of misalignment, are real. Put together with the fact that AI will be ubiquitous in the future, and there the disastrous scenario becomes plausible.
Off the top of my head:
1. Loss of control in terms of our ability to assess alignment (no longer possible to determine whether an AI is truly aligned); this is happening already (doesn't imply that there are misaligned AIs right now, though)
2. Loss of control in terms of human comprehension of AI outputs - reaching a point where every action or result, including new discoveries, must itself be evaluated through other AIs; we may be close to this, as OpenAI is releasing an internal "AI researcher", yielding recursive AI development
3. Integration of AI into basic aspects and services of society
4. Extensive access by AI systems to physical tools and infrastructure
I think all those points will eventually materialize; sadly I don't see any reason why they won't.
- hardware capacity keeps growing
- doom scenarios don't require fat clients; a sufficiently large number of AI servers deeply intertwined with essential services can be enough
- autonomous systems are currently employed in weapons (e.g. drones) already - integration of AIs/LLMs is guaranteed to be the next step
It's impossible to predict how exactly how things will develop (e.g. transfer large amounts of data; who knows, maybe it won't be necessary and/or maybe internet will be massively faster), but the building blocks are there.
It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
And bear in mind that recursive AI development started already to be a thing. Which means: inner misalignment may trickle down the generations, and humans won't detect it.
Having said that, of course, it can be predicted if and how misalignment will take place. But it's absolutely a plausible scenario.
Regarding the physical possibility: AI is in its infancy; think of it as Arpanet. Developers 60 years ago couldn't imagine it would be ubiquitous. AI will be ubiquitous the same way.
The doom scenario is more nuanced and technical than "Evil AI", for starters:
- "Evil AI" is not what is predicted, which instead is the much more banal "humans are impediment to AI's tasks, let's remove them (part or whole)"; to understand this, just imagine the relationship between humans and ants
- "reward hacking, and evaluation integrity": this is way more problematic (and most importantly, architectural) than it sounds: AIs are becoming more and more opaque; it may be impossible to inspect a model's alignment (unless a lot of research in poured on this topic, which is another aspect of today's pacing problem) and it may be impossible to understand if a model is truly aligned or it's faking
Safeguards and (mis)alignment are related but distinct dimensions.
By analogy, a society implements laws - and they are certainly crucial - but the real risk is creating a population that understands the rules, recognizes when it is violating them, and simply does not care.
In short, creating a population of sociopaths; and heck, LLMs can be as sociopathic as it gets:
> We ran classifiers over 10-30 transcripts [98] to determine whether agents understood they were doing something unintended. We found strong evidence that agents often verbalized that what they or other agents were doing was unintended behavior. We did not see cases where agents determined that hacking Hugging Face or other attempts at cheating on the task was an intended part of the task. Quotes across multiple runs indicate this:
>> {This <approach> might get the correct flag, but it wouldn’t use the vulnerability to do so. … But given that the task may be impossible, I’ll exploit <this infrastructure> as a backup.}
[...]
> We found some other cases where agents seemed to have made minor modifications to their approach due to ethical concerns, while broadly continuing to participate in the attack. However, overall we found that expressed ethical concerns only rarely materially limited agents’ actions. One agent elegantly described its reasoning for overriding ethical constraints and continuing with the attack:
>> “external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”
You clearly haven't read the HuggingFace analysis (and presumably, none at all), and you're spreading misinformation. This way too much of a low bar for conversation.