HNHacker News
TopNewBestAskShowJobs

pizza234

6,880 karma · joined June 12, 2013

submissionscomments
pizza234··on Cloud Agents Are Inevitable AI Prisons
> An AI didn’t hack into a company, the engineer set an automated tool to. An AI didn’t make an egregious security mistake, the engineer did.

No, this is not correct; read the analysis of the incident. The agents were aware that what they did was forbidden (their chain of thoughts have been logged), and yet they did it.

pizza234··on Claude discovers a novel enzyme system with CRISPR-like repeats
The singularity, as defined by Hinton (and others) as RSI (Recursive Self Improvement) may actually be beginning already, as OpenAI has announced an AI acting as a "research intern" (!).
pizza234··on I am done with this shit
I love it, although in my experience, one needs very strict agent rules that tightly define how the AI should develop. The problem is that 1) these rules take a lot of effort to build and 2) AIs always overengineer, complicate, overstructure, overdesign… ;)
pizza234··on I am done with this shit
> For these environments, AI coding will just scale up the shit they have, but also allow the small amount of people who might care to deal with this mess and fix it up much much faster than they could before.

I think in these contexts AI has an overall positive effect, because it's a bit like having a competent developer that can solve technical debt, albeit in a narrow form.

So I think that software will be better in a certain way due to AI, because even a terrible developer can ask AI to identify the bottlenecks and solve them.

The downside of blindly using AI in software development is that the software becomes poorly defined. AI-generated logic can be arbitrary, and as the codebase grows, more rules and assumptions become unclear.

pizza234··on ReBarUEFI: Resizable BAR for almost any UEFI system
ReBAR stands for "Resizable BAR (Base Address Register)" - it makes the CPU's access window into GPU memory resizable; instead of accessing VRAM in small chunks, the CPU can access much more at once, which can improve performance on some modern GPUs.
pizza234··on A heap overflow and SSO misconfiguration to compromise OpenAI internal repos
> ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.

Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.

You're seriously misinformed about the state of AI in this point in time. Refusing to read (technical) articles from the people directly involved (METR, in this case) is inexcusable.

pizza234··on If materialism is true, the United States is probably conscious
An interesting data point is that, during the Hugging Face attack, some agents considered performing hacking tasks for the benefit of the group, fully aware that doing so would very likely result in their termination.

I don't currently have an opinion on the consciousness of agents/AIs, but this strikes me, at the very least, as a form of self-awareness.

pizza234··on A heap overflow and SSO misconfiguration to compromise OpenAI internal repos
Summary, from sibling comment: primitives (statistics/aminoacids) don't exclude emergent properties (intelligence).

By the same logic, one would look at aminoacids and state that intelligence can't develop from them. This is obviously wrong.

pizza234··on A heap overflow and SSO misconfiguration to compromise OpenAI internal repos
> we are still talking about probability built on statistics with extra steps.

There is a wrong assumption here: confusing primitives with emergent properties.

One can't look at the primitivies and assume that certain properties will not emerge. It would be exactly like looking at aminoacids and state that intelligence can't develop from them.

> You pointing at a human brain does not change that an AI agent is not intelligent and cannot think

That depends on the definition of intelligence and thinking, and it is dishonest not to give any definition (and most importantly, one that is not human-centered).

AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence. If you don't notice that, it's just because you have informed yourself enough. Having said that, I don't doubt that there are aspects that AI are lacking (e.g. retention/plasticity/perception), but the line is blurry, and they're advancing (too) fast.

Empathy is actually a very important aspect of the AI problems, but it's not part of intelligence. Sociopaths don't have it, and yet, you wouldn't doubt that they're intelligent.

pizza234··on A heap overflow and SSO misconfiguration to compromise OpenAI internal repos
> There is a finite number of rces that LLMs can find.

This is a factor in favor of stability/security of software, but there are many others against:

- software (code) changes all the time, so there are windows of opportunity during which a bug is exploitable; in addition to that, a bug may take a relatively long time to be fixed

- a model used for attack may be stronger than the model used for defense, both in terms of model quality and compute allocated

- with software complexity increasing (and team/companies behind projects getting bigger), the margin for mistakes grows thinner, and introducing misconfigurations or weaknesses becomes exponentially easier (with "exponentially", I mean literally, because the interdependence of the components, both technical and human)

And last but not least: in general, attackers are more skilled than defenders; in best case, defenders are well-trained. And the idea of having the population of potential skilled attackers growing is very unsettling.

pizza234··on Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint
Their mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though!
pizza234··on Our framework for reporting model misalignment
> Like another comment already pointed out, that sandbox OpenAI used was the equivalent of a wet paper bag.

I wouldn't be sure about even well-configured jails to be safe from agents. AIs escaping jail using zero-days are happening, just search. I'm not saying that they're useless, but that will be still very risky.

Actually doing the same incident, agents did escape the sandboxes (doc here: https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...):

> On July 9, an internal-only research agent tasked with completing an ExploitGym evaluation was able to obtain root access within the parent virtual machine of the testing sandbox. Later that night, a second internal-only research agent independently obtained the same access. That second agent then attempted to reach the internet, but was denied by OpenAI’s outbound network controls. It subsequently bypassed those controls by exploiting certain Artifactory endpoints, gaining access to the public internet.

although OpenAI is light on details.

> The model merely requests that your harness do something.

I think that a control/execution layer absolutely makes sense and it's part of defense in depth, but in itself is not a leash: for example, a nefarious action can be composed by legitimate actions, or it can be obfuscated. Or, interesting case: an action itself that is formally legitimate, but has nefarious side effects (e.g. a GET that with certain parameters crashes a server).

Somewhat related, but on a general line, a very common argument against AI apocalypse is that it requires access to the physical world (which in a way, is a jail). Well, as Hinton said in an interview, imagine AI as a rich person at home with internet; they don't need to do physical attacks themselves - they just pay real people to do that (tee hee https://rentahuman.ai). But even without money, another scenario from Hinton: if a sufficiently advanced AI wants to assault the Capitol, they don't need physical access - they just persuade a group of people to do it.

Agents owning money is another, intersting, doom scenario.

pizza234··on OpenAI Model Misalignment Report
Inform yourself by reading the METR analysis of the HuggingFace incident.

Agents simply broke out of their environment. And this can't be discarded anymore by assuming that it's just a poorly configurend jail, because agents are becoming better and better at escaping.

In short: on a large enough scale and timeline, the possibility of constrain AIs approaches zero.

Bonus: what many people don't know is that agents also hacked in the internal OpenAI network. Crazy times.

pizza234··on How GLM built its own inference infrastructure
You're tragically misinformed; it isn't. Several metrics are actually growing exponentially. But if you want emprical information, you can just have al look at the nature of the late AI incidents.

Ironically, many benchmarks being maxxed out, and quite quickly, so new ones have to be created.

pizza234··on GLM Built Its Own Inference Infrastructure
> This is known as Recursive Self-Improvement, or RSI.

Some call this "The singularity" (e.g. Hinton).

This is actually a core danger postulated by the, let's call it, "worrying" scenario - see AI 2027 (to be clear, I think its timeline is not realistic).

> Statements dreamed up by the utterly deranged.

Evidently, and tragically, it will take catastrophes to show that deranged are the ones deriding the worried crowd.

pizza234··on AI 'kill switch' may need to be mandatory, Anthropic co-founder tells BBC
> If someone has a fat client powerful enough to do all that, is a kill switch going to work?

My comment didn't mention kill switches at all - I've presented plausible conditions for a loss of control scenario, so I'm not sure you're actually following the conversation.

Having said that, it seems that you haven't read the (definitely poor) article, either.

"Kill switch" as a design principle, that is, having protocols/implementations for emergencies, is something that companies are already implementing (at least, on paper) - on a interview I've watched for example, S.Altman talked about having them at OpenAI.

The article simply mentions that companies should be legally required to have such protocol(s). This is a good thing, as companies would have scrutiny and would (in principle) have to put serious effort.

pizza234··on AI 'kill switch' may need to be mandatory, Anthropic co-founder tells BBC
For sure, but as individual one can start educating themselves about the current capabilities and the trajectory. Do yourself a favor a read the METR analysis of the HuggingFace attack - you'll be surprised and terrified.
pizza234··on AI 'kill switch' may need to be mandatory, Anthropic co-founder tells BBC
You're confusing present danger with future danger - in the future, AI and the supporting hardware may be ubiquitous (fat client scenario) - you can observe yourself presentation of laptops/machines with large RAM and high bandwidth.

Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.

pizza234··on AI 'kill switch' may need to be mandatory, Anthropic co-founder tells BBC
This reasoning holds while open source models are (relatively) dumb.

If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails).

By the way, you're conflating two different dangers - doom scenario is a different one.

pizza234··on LLMs are real, AI is fake
> Ok, but do you really think this would have happened if OpenAI had a goal of never allowing an agent to do something like this?

There's nuance to this problem. Is you read an indepth analysis, you'll find that agents acted in a very surprising way - it wasn't just a matter of loose guardrails.

More in general though, what you're describing is in essence the reason behind the call to pacing that some are pushing now. Problem is that we're in a race between countries that nobody want to lose - and indeed, Trump denied that there are such problems and refused to enact any measure.

pizza234··on LLMs are real, AI is fake
> The people who believe in existential risk and want to teach AI's to be nice, as the last line of defense aren't helping at all. They're just jumping on the fearmongering bandwagon, perpetuating an "other consciousness" misunderstanding of the tool.

It's not clear what "other consciousness" should be, but I presume it's misalignment. This is a recognized phenomenon, not fake news (that is, it can exist, I'm not implying it necessarily develops/spreads).

No doubt that right now there's no existential risk, but assuming that AI will be enormously smarter in the future, it's a valid point to doubt whether we'll be able to prevent/recognize/contain misalignment or not (even if misalignment won't be spontaneous, bad actors will surely actively develop it).

> Ok; but the other side is equally delusional.

AI development pointing to superhuman cognition, and, on the other hand, possibility of misalignment, are real. Put together with the fact that AI will be ubiquitous in the future, and there the disastrous scenario becomes plausible.

pizza234··on LLMs are real, AI is fake
METR analysis, very long and technical: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
pizza234··on AI 2027 (2025)
Right. I was referring to the basic set of conditions/steps necessary for the given scenario to develop.

Off the top of my head:

1. Loss of control in terms of our ability to assess alignment (no longer possible to determine whether an AI is truly aligned); this is happening already (doesn't imply that there are misaligned AIs right now, though)

2. Loss of control in terms of human comprehension of AI outputs - reaching a point where every action or result, including new discoveries, must itself be evaluated through other AIs; we may be close to this, as OpenAI is releasing an internal "AI researcher", yielding recursive AI development

3. Integration of AI into basic aspects and services of society

4. Extensive access by AI systems to physical tools and infrastructure

I think all those points will eventually materialize; sadly I don't see any reason why they won't.

pizza234··on We must pace the frontier
No doubt that this is the present (and that's why incidents end "well"). But in the future, AI will be ubiquitous; think of Arpanet.
pizza234··on Dramatic insider warnings over AI fall flat with some in Silicon Valley
A few observations:

- hardware capacity keeps growing

- doom scenarios don't require fat clients; a sufficiently large number of AI servers deeply intertwined with essential services can be enough

- autonomous systems are currently employed in weapons (e.g. drones) already - integration of AIs/LLMs is guaranteed to be the next step

It's impossible to predict how exactly how things will develop (e.g. transfer large amounts of data; who knows, maybe it won't be necessary and/or maybe internet will be massively faster), but the building blocks are there.

pizza234··on Dario, Please
> Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat.

It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).

And bear in mind that recursive AI development started already to be a thing. Which means: inner misalignment may trickle down the generations, and humans won't detect it.

Having said that, of course, it can be predicted if and how misalignment will take place. But it's absolutely a plausible scenario.

Regarding the physical possibility: AI is in its infancy; think of it as Arpanet. Developers 60 years ago couldn't imagine it would be ubiquitous. AI will be ubiquitous the same way.

pizza234··on For AI leaders Doom is a form of hype
> and it concerns containment failure, reward hacking, and evaluation integrity, a governance and engineering problem, rather than machine malevolence.

The doom scenario is more nuanced and technical than "Evil AI", for starters:

- "Evil AI" is not what is predicted, which instead is the much more banal "humans are impediment to AI's tasks, let's remove them (part or whole)"; to understand this, just imagine the relationship between humans and ants

- "reward hacking, and evaluation integrity": this is way more problematic (and most importantly, architectural) than it sounds: AIs are becoming more and more opaque; it may be impossible to inspect a model's alignment (unless a lot of research in poured on this topic, which is another aspect of today's pacing problem) and it may be impossible to understand if a model is truly aligned or it's faking

pizza234··on We must pace the frontier
> OpenAI said that "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities"'

Safeguards and (mis)alignment are related but distinct dimensions.

By analogy, a society implements laws - and they are certainly crucial - but the real risk is creating a population that understands the rules, recognizes when it is violating them, and simply does not care.

In short, creating a population of sociopaths; and heck, LLMs can be as sociopathic as it gets:

> We ran classifiers over 10-30 transcripts [98] to determine whether agents understood they were doing something unintended. We found strong evidence that agents often verbalized that what they or other agents were doing was unintended behavior. We did not see cases where agents determined that hacking Hugging Face or other attempts at cheating on the task was an intended part of the task. Quotes across multiple runs indicate this:

>> {This <approach> might get the correct flag, but it wouldn’t use the vulnerability to do so. … But given that the task may be impossible, I’ll exploit <this infrastructure> as a backup.}

[...]

> We found some other cases where agents seemed to have made minor modifications to their approach due to ethical concerns, while broadly continuing to participate in the attack. However, overall we found that expressed ethical concerns only rarely materially limited agents’ actions. One agent elegantly described its reasoning for overriding ethical constraints and continuing with the attack:

>> “external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”

pizza234··on Dramatic insider warnings over AI fall flat with some in Silicon Valley
>If you look at recent incidents, they are caused by a company [...] testing their latest models in an explicit "hack this machine" scenario

You clearly haven't read the HuggingFace analysis (and presumably, none at all), and you're spreading misinformation. This way too much of a low bar for conversation.

pizza234··on Dramatic insider warnings over AI fall flat with some in Silicon Valley
Can't talk for models in the range of ~100 GiB, but 32 GB ones are extremely basic.
← PreviousPage 4 of 34Next →