Comment by pizlonator | Hacker News Reader

Comment by pizlonator | Hacker News Reader

Parent Full thread

It’s true that I’m an AI adopter. My psychosis is pretty advanced. I’m running four clankers in parallel right now.

The 9 bugs I got were clear security bugs under the Fil-C definition of memory safety. One reason why my experience could be an outlier is the effort I’ve gone to defining what the security properties of Fil-C are, to the point that there’s a crystal clear falsifiable boundary. So the bugs aren’t a matter of opinion or judgement. That helps clankers and it helps me when screening the bugs.

> concretely nothing of any significance has happened

This is a good point, but “cyber apocalypse dooming is an exaggeration” is a different statement than “bugs found by Anthropic aren’t real”. I agree that the dooming is an exaggeration. I don’t agree that Anthropic’s bugs suck.

Two reasons why bugs-found-by AI-are-really-scary and nothing-is-happening can be true at the same time:

1. Anthropic is sharing the bugs found by their most capable model with defenders, not attackers. Defenders are working overtime fixing those bugs before the embargo runs out.

2. The attackers that have access to Anthropic-class capability are using their bugs surgically. They aren’t using them so widely that folks would notice. That strategy is business as usual for advanced attackers

I obviously don't know if this applies for the Fil-C bugs but what we saw in Rust is that there's plenty of "technically that's not OK" which is very distant from "this will actually cause real world problems" https://forge-blog.github.io/posts/rust-std-memory-safety/

For example it's definitely not OK that you can impl Pattern in a way that causes non-UTF8 data in a string with safe code. But you probably didn't let random attackers impl Pattern, on a nightly Rust where that's even possible, and so that's not going to cause problems in our real world - even though it's wrong and must be fixed.

I'm speculating out on a limb here, but maybe the Mythos zero-days that were developed into working exploits involved a few of those "usually won't cause real-world problems" happenstances.

A model with all the world's knowledge might know exactly how to weave them together into a successful exploit. And if tomorrow's models are even more capable, perhaps what we're experiencing now is just a fire drill for what's coming.

Most of the bugs were things that fall into the “won’t cause real world problems” category.

But they were all things that unambiguously violated the Fil-C promise

> One reason why my experience could be an outlier is the effort I’ve gone to defining what the security properties of Fil-C are, to the point that there’s a crystal clear falsifiable boundary.

Building software boils down to defining things correctly. So, it sounds like you did most of the work here. Imagine that you took your definition and moved it into a formal specification (programming language) and then ran that to verify the security properties of your system. What's different?

Or, imagine that you took your original code as your specification language and then iterated on your understanding of it until it was correct. What's different?

Attention highlights and resolves bugs, so that's the baseline. In order for AI to get value, it has to be some kind of multiplier on your attention, but the reality is generally the opposite.

Reply on news.ycombinator.com