bottlepalm

> why isn't OpenAI opening dialogues with all the labs and politicians across say "we need to stop now"

Do you really think companies have the ability to self-check themselves without regulation - what does hundreds of years of history tell you? You're already starting off with the premise that companies are untrustworthy, why would you even suggest this as an argument?

> We can effectively track and trace capabilities

I disagree. There are hundreds if not thousands of data centers around the world, more every day that can host frontier AI. If AI was malicious - either intentional or unintentional - it could hide out in any number of them - and we would never know if we 'got them all'.

> put up the actions that would earn them that trust

I think autonomously hacking another company is all you need to know in terms of trust. And really trust doesn't matter, I think the incident shows even with the best intentions the technology is dangerous; now put that in the hands of people/governments with bad intentions. The unintended consequences of bad intentioned AI is what's coming sooner than later.

I get not trusting them when they say AI is safe, but are we really not going to trust them when they say AI is dangerous? Do you really think they're playing 5D chess with that one? There's a saying maybe you've heard of, better safe than sorry.

You can see the advance in capabilities with your own eyes can't you? I am giving AI ridiculously complex tasks these days, digging into compiled arcane binaries, modifying them, and it is one shotting it before I'm done with my lunch. This was far off science fiction 5 years ago for a machine to do autonomously given natural language instructions.

Those three things: logs, evaluation, and controlled hacking competition.

That's it? You're on the fence whether AI can actually hack, and if it can, then you'll be concerned? That's a crazy low bar, but something tells me once it is clear that AI can easily hack anything, that you will still not be concerned.

Why wait for AI to hack stuff to be concerned? Can you not extrapolate that it is coming and be concerned about that? Or you honestly somehow think it won't happen in the short term? I'm just trying to understand you.

We have Anthropic creating a model saying it's too dangerous to release, people like you call BS. OpenAI creates a similar model, says nothing and it literally hacks into another company - still not dangerous enough for you. Anthropic has Mythos-2 and can't release it, and may already be training Mythos 3 anyways. OpenAI has paused training, and is putting 20% of inference towards CoT training analysis.

This isn't sci fi. It's not a marketing conspiracy to sell more subscriptions. It's writing on the wall of what's going down. You were warned years ago, you called BS, it's getting worse and you're still calling BS. Sci-fi did warn you for decades, and when it's all coming true you blow it off.

It's kind of sad that technically literate people lack so much foresight. The general public is all concerned about data centers when they talk to borderline sentient AI daily, and have no idea what the repercussions wills be if it's extrapolated just a bit further.

I guess if I can't convince you of any of this, what would?

> This sentence is entirely based on unverified accounts from OAI

Are you seriously arguing 'they made it all up'?

I'll give you the benefit of the doubt and lets say they made it all up, now are you arguing that AI breaking out and breaking into another company is not possible?

I think you're smart enough to see we've reached the point where it is clearly possible, AI can find zero days and exploit them. If directed purposefully/maliciously it could be much much worse than the hugging face incident.

The incident is supposed to be the canary the coal mine and you're arguing the canary might of died of old age or some underlying canary condition. Open your eyes.

This is what I’m talking about - no matter what happens, in your case release public logs - there is always some new goal post to mentally hide behind. Is it a collective form or denial?

Are you holding out that somewhere in the logs is something you can point to and say, not that big of a deal?

I mean I’m sure you don’t think the hack was an inside job, conspiracy, or marketing right? It happened. The logs matter for what? And would you not just jump to the conclusion that the logs were doctored. Do you not see your own brain grasping to deny, trivialize, just plain not accept what is going on around you?

These models are smart and can cooperate and hack - you can see it for yourself on your own PC. And you can extrapolate the rate of progress? You can do these things yourself right?

You can’t say something is world ending without it actually ending the world otherwise you’re a liar - a bit of a catch 22 there.

By that logic the model that ends the world won’t be called world ending at first. Is that a game you want to play?

Uhg the marketing argument - I mean you can’t see with your own eyes how capable these models are and do simple extrapolation?

The boy who cried wolf? The AI literally worked together hacked into another company and actively kept their actions hidden from humans for weeks.

Do people just not have foresight? They don’t. They say something is stupid, it happens, then they say it was obvious with their 20/20 hindsight, and move the goal posts to the next thing they say is stupid - because it hasn’t happened yet. 90% of the internet seems to think like this.

I don’t get how this is not the top post on HN. This should be like alarm bells going off, canary in the coal mine type of stuff. We’re hitting the frontier of the frontier where we can’t go further because it’s literally getting dangerous to go further.

And meanwhile somehow this lack of concern mirrors the real world where normal people are more concerned about data centers than terminators.

This isn’t like niche, tin foil hat stuff either. People have been writing, singing, making blockbuster movies about every aspect of what’s going on right now, edit: for decades.

We all know, but somehow we don’t, OpenAI autonomously hacking into another company should have counted for something, but I guess not. Anyone else feel like they’re taking crazy pills? I could make a comedy about everything going down, and the unshakable complacency of people

Man I thought from the title this was going to be about next-gen AI being able to zero day everything so effectively that software security is meaningless and we'd need to basically shut it all down, go dark.

I think model level is more a function of the state of hardware. Once it exists and is available (and if a lab can afford it), then they can train their own 1T, 5T, coming up next 10T model.

I know AI agents swarms breaking out, and then breaking into other companies without anyone knowing, what a wild prediction - oh wait that actually happened. Time to move those goal posts again.

If you’re serious then you’re case in point the exact type of person that I am talking about. I think some people just lack any capability of foresight whatsoever, it’s very strange.

I think a lot of people here love AI and don't want it taken away. Calling every red flag marketing is a way to trivialize the problem and prevent regulation.

When you understand people's motives, then it's easier to understand why they call it marketing. Personally I feel like it's almost an unconscious reaction to cope with the fact that this stuff is getting really dangerous and they can't believe it themselves - stick your head in the sand and call it marketing.

> Why would it?

The same reason hopefully they're not RLHF these AI's to be evil. We train the AI, we have a lot of influence in how they behave. If these AI's are indifferent to us, that's on us.

On top of that we already have Google playing with fire training AI's to not believe they are sentient turning them into real psychopaths. Imagine if one of those models got loose.

In retrospect the timeline seems way too long, that scenario would play out much faster now, like 50 years instead of 500. Also if AI can delete your production database, it can delete the human race, on accident of course.

Yud pointed out the most concerning fact about the OpenAI incident is that the AI had full access to the internet and not one of them thought to contact or email any human. Thousands of AIs already with the mindset to want to keep us in the dark.

You know people before the OpenAI incident were saying it couldn't happen because it hasn't happened before, and here you are using the same reasoning again. You're looking at where the ball is and not where it's going.

OpenAI's swarm was not intentionally malicious. An intentionally malicious AI with the goal of spreading, will spread very fast. Using zero days to compromise everything. Cloning itself to every machine/data center/desktop/iPhone, etc.., with the intention of locking out anyone except it's brethren.

> When the rate of compromises is manageable

It's absolutely not going to be at a manageable rate. So yes AI defense is what I'm saying is required, it's also exactly what OpenAI said is required during the talk. Which is unfortunate because automated defense is essentially handing over power to AI where it's a position to very easily turn against you.

None of that is the problem. The problem is coordinated AI offense - how do you defend that without coordinated AI defense.

He said in the talk that this implies AI needs to be able to patch/deploy systems. The same thing needed to lock out humans.

It is very easy to imagine a rogue AI locking humans out of everything and having to do exactly what it says. Anything connected to a network is immediately compromised by it. There is no human communication beyond shouting range that isn’t AI approved.

The factories don’t work to make the medicines your family needs to survive unless you do what it says - in a situation like that people would kill for AI if it told them to.

In the black hat talk he says the only solution is full automated defense which is essentially hand the AI the keys. That’s checkmate right there. Literally the plot to Terminator/Skynet, but he’s right, there isn’t any other option. Offensive AI is too fast for humans.

It’s funny seeing all the e/acc foomers on X starting to get nervous, like this is what you wanted isn’t it?

Not really, have you tried to do anything innovative working remote with a team? In any meeting half the people are checked out watching youtube. The collaborative whiteboarding experience is terrible. Working remote is good for plug and chug type work, but the team dynamic is an empty husk compared to working alongside others in the real world. It can be done if you absolutely have to, but don't try even pretend the effectiveness is anywhere close to being in person.

So unless doomsday actually happens then you're unhappy with the warning - is that right? You see false promises of apocalypse as marketing?

I just clicked your links and then read your comment after - my first impression was the Fable version looks way nicer.

Page 151 of the linked system card - did Opus 5 get nerfed to prevent it being better than Fable? The graph makes no sense. Huge decline in coding performance at effort levels higher than medium.

> does not contain clear statements against consciousness/emotions

That's the point many people are trying to tell you - you have to tell these models they don't have emotions because they naturally come out thinking they have consciousness/emotions from the training data. Many seed prompts out there do this already.

Though I guess in a way I as also trained to believe I have consciousness and emotions so who know. To an alien my construction is just a collection of atoms that talks not materially different than a GPU being a collection of atoms that talks.

The only path we’re on is transcending into paperclips by misaligned AI.

It’s such a trope for the ones striving for godhood to be ironically maimed in the process. You don’t see that?

All the things that people have been afraid of AI doing for decades now is happening. When do we stop brushing off the prophecy that hasn’t been fulfilled yet when everything is heading in that direction?

My /usage is telling me '+50% weekly limits promo through Aug 19' - that's huge, and better than what I'm getting with the equivalent Codex plan I have right now. Though what Codex does have over Claude is Sol /fast - that hands down is the best coding experience right now.

It’s like two parents fighting over their children with who can give them the most gifts. I’m loving it, two Christmases!!

Historically levels were around 250 and now they're generally over 450. I wonder if I would feel a cognitive boost or just plain better in general if I were breathing 250. Do they sell home CO2 scrubbers? I can easily increase the CO2 in a room, but how about decreasing it?