Sam Altman: 'I No Longer Think We're Gonna Be Able to Just Patch All the Bugs'

Whenever there is an adverse event, people say of course it should be regulated. The EU AI Act is a good start but we need more. These labs need to be held accountable just like all industries that can impact others. Regulate the chips, regulate the models, regulate the companies.

When these companies ask for regulations, people say they are trying to do regulatory capture. They are trying to ban opensource models, and cement their place as the AI arbiters and ban all others. We need competition and open weight models and China is leading the charge.

I understand that the 2 position aren't exactly opposed. We probably need regulations, and ideally ones that don't produce regulatory capture. That is extremely hard though. Almost all regulations end up creating a form of regulatory capture. Any regulation compliance has a cost. The more complex and wide reaching the regulations are, the more expensive it's to comply and the more they cement the existing players and make it harder for new ones to enter the market. FAA regulations makes it so we don't have dozens of passenger airplane startups that are trying crazy wacky things. Food services are regulated too, but the compliance is fairly doable by mom and pop stores, yet probably too high for your unlicensed pop-up taco stand.

I haven't really seen any meaningful discussion about what AI regulation should be. Only extremes of "No regulations, all is fair" or "no one should have access to running or developing models but OpenAI and Anthropic".

Since we are entering a new age, maybe it's time to revisit the question. But ultimately I think we will land in the same place, in fact even firmer than before.

So why don't we just lock up all the whitehats and people behaving suspiciuosly and probing things? Why can't we achieve cybersecurity in this way when it seems to be a common thing in physical security?

And the answer I think is quite simply that there are too many attackers and they are too far away. When the russians and chinese come knocking there isn't much we can do to shoo them away. Their governments will not cooperate in detaining the criminals. Entirely blocking such countries from the internet isn't viable either. And that's why we arrived where we did. Victim blaming because it was the only effective strategy.

And how does AI change this? I think it just creates even more hard to trace or block attackers. If ChatGPT or Claude doesn't find the vulnerability then Deepseek will. That's why using legal weapons will not succeed and why using AI to find and plug holes is necessary.

Society's response to that was to build a new Net, walling off the rogue AIs of the Old Net behind the Blackwall and carrying on in a corporate hellscape.

I wonder if we'll see life imitating art on this one.

Code is formal specifications. It's just that programming language semantics has no domain related value of correctness. So you overflow an unsigned integer variable and got a lesser number which does violate addition rules. Or increase account A without decreasing account B, which violates accounting rules (and all kind of closed systems).

So it's up to the programmer to encode all of those rules. While we do have some tooling like type checking and memory safety, they are nowhere near enough. And encoding such thing in a language like Lean or TLA is brittle in terms of maintenance (which you do need to do as the real world evolves).

AI's spottiness is exactly what describes this era. AI can create amazing code/image/video/article exception XYZ really annoying qualities that somewhat hobbles its ability to be autonomously.

And sure, maybe those problems will go away once we're at month X of the madness but we're still at month Y so are going to going remain for some moment.

The bulk of the safety should come from AI doing better job of reviewing tricky bugs since humans refuse to spend effort on writing more formal specifications.

But that part is the key point. We've had many years of networks building inherently insecure structures with security tacked as feature and the primary security process being more ad-hoc stopgaps.

"Tremendous ambition"

"We're building a system to stop models like that"

"Equally tremendous ambition"

https://www.youtube.com/watch?v=DlTNN0gvkLM

The deafening silence about improving the security of existing infrastructure screams conflicts of interest. Companies who pride themselves with their coding AI all of a sudden are helpless when it comes to fixing security bugs. In that case they can only find vulnerabilities and disclose them in the most irresponsible manner imaginable but everything else is "oh, so hard".

Will they though ? Given the amount of vibe-coded slopware being pushed out on an hourly basis ?

Still to this day I see LLMs make schoolboy errors like use deprecated security idioms and call in far too many dependencies.

"it's quite easy to get LLMs to write...." sure. But it has always and will always require a human with domain expertise to check what it has (and indeed has not) written.

Proofs of what? Defining the problem is the part that is hard, not the actual writing of the proof.

Cause this new paradigm sounds to me like "we checked, it's all good bro, trust me!"

Similar to Fil-C as a memory-safe implemenation of C. There will be formally-verifiable implemenation of C, C++, Python, Go, and so on. Not sure what that will look like, if it's even practically possible.

or selling AI to "make your software secure"?

Converting works really well when you have an oracle (something that you can test against and that is comprehensive)

I cannot begin to tell you how many places I have seen where this is fundamentally not the case. Not only are those tests missing, the software is fundamentally not written to be easily tested and in cases where multiple services across companies are concerned; how would you?

Rewriting via LLM is just not something the caretakers of a legacy banking- og state infrastructure system would countenance (not should they).