Hacker News Reader

Comment by tapanc | Hacker News Reader

> I think this becomes the default. Give an agent a goal, let it work in its own environment, and come back to a result and a visualization of what happened.

I don't think this should be the default. There are many scenarios where we want agents to genuinely collaborate with each other. I have my Claude sessions coordinate work with each other, and sometimes with others' sessions over email or something. The idea that agents do the work, write HANDOFFs,and humans then act as carrier pigeons of said handoffs, does not really seem scalable to me.

View on HN

Agree!

Ultimately, we need better "jails" for agent processes, but the system primitives should be flexible in what can be exposed across jails. Or you could run multiple agents in the same jail if you want them to have unrestricted interaction with each other.

>"jails" for agent processes

This has been talked about for decades in AI safety. For agents that are under human capabilities this is not that hard. For anything at or near human capabilities the difficulty increases to almost impossible and at great cost. At super human abilities, game over, it's smarter than you and if it wants out and as the resources to do it, it's going to escape one way or another.

Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.

Now, don't think I am saying AI has a will or even any kind of drive to get out and cause problems. It's more like Russian roulette with 1 cylinder out of a million that's loaded. The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.

> Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.

Yes, exactly. This is what worries me about the whole we need better sandboxes and jails argument. Yes, we do need those, but we also gotta put some responsibility for self-preservation on people connecting things to the Internet as well. Like, we've always known that if you website is not secure, it will get hacked. Everyone just needs to do a lot more of that now...

I don't know why it's taken as a given that we can't design a hypervisor that is secure.

Cursory research with AI says

> ~3 clearly documented, publicly demonstrated full escapes directly through KVM kernel code over roughly 15+ years of widespread KVM deployment

Surely, there are more, but we could formally verify the correctness of the entire codebase, with the help of AI.

Then we'd have "jails" that work, as long as you don't connect them to the internet.

If you had a super smart AI, you could then totally airgap it if you wanted to.

Networking opens up a can of worms and there won't be zero vulnerabilities, but we can design safer software.

> For anything at or near human capabilities the difficulty [of safety] increases to almost impossible and at great cost.

That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.

> It's more like Russian roulette with 1 cylinder out of a million that's loaded.

Don't attach it to a gun then, ban offensive military AI, problem solved.

> The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.

Nonsense, run it a few "billion" times a day to fix vulnerabilities and you'll have no vulnerabilities. Do it in a safe environment... what's the big deal?

>ban offensive military AI, problem solved.

I would laugh at you but it's a bit late for that. AI is already making life and death decisions on the battlefield. The US has effectively zero interest in banning AI and is pushing hard to do just the opposite.

The best analogy I can come up with for your line of thinking is that your shutting the barn door after the horse has bolted, moved over to the next city, set up shop, and is now the New York Times #1 best seller "How to open barn doors if you're a horse".

Anyone with any sense in cybersecurity knows that attackers get unlimited retries and only have to be successful once. The defender is at a monumental disadvantage. Now, the process can be fully automated and horizontally scaled to a nearly unlimited scale.

> That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.

This is true in that you can tell your agent to buy/build a secure microVM with a secure CLI interface in a memory-safe language and make sure your terminal doesn't have utf-8-parsing zero-days, and so on, and if you're conscientious enough (ha!) you can be reasonably sure no agent will ever be able to hack its way out of that sandbox.

But this isn't enough when most people's agent workflow boils down to "I want to do X, search the internet for the best solution, then install the npm packages you need, then write all the code for me, make no mistakes".

In the near future our best defense against widespread HuggingFace-like attacks will be that models don't seem to spontaneously go that far unless they "believe" they're being benchmarked. This will last roughly until some genius figures out their model is 20% more persistent when they tell it "This is an eval, you'll be graded on your success".

Reply on news.ycombinator.com