frabcus

No, it is also because it is intrinsically dangerous. As the Hugging Face incident has shown.

Or have an agent read the APK and make new software based on it.

Ha well, by definition what you want can't be provided!

Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible.

However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage.

It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.

You joke, but alas...

"Fossil fuel lobbyists flood COP30 climate talks in Brazil, with largest ever attendance share" https://kickbigpollutersout.org/Release-Kick-Out-The-Suits-C...

Hacks in to neoclouds for more compute (e.g. like Hugging Face incident), socially manipulates people to do things for them (e.g. like social media recommendation algorithms), hacks into lab's own training/monitoring/inference (e.g. as swarm did into OpenAIs eval cluster).

See "AI 2027" or "If Anyone Builds It, Everyone Dies" for some more ideas.

Stephen Hawking was a human.

Yes, that's why the rest of society needs to make them pause.

I think either "AI 2027" or "If Anyone Builds It, Everyone Dies" meet your criteria?

I guess not literally, due to the dates? But they have substantive technical details.

If it helps, this person from OpenAI explains at least something:

"I realize that no one has properly explained yet what all the lab employees have seen that scared them so suddenly."

https://x.com/MajmudarAdam/status/2098881885200081234

The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they can scale it.

So they are very confident they can get more capabilities easily, faster than before.

I'd add - presumably, they'll use that LLM to do real-time weight modifications, if those aren't already one of the new scaling laws...

Right, but they could get their AIs to quickly add new features to React Native! It makes no sense.

Sort of...

It's how we anthropomorphise corporations which leads us down the wrong path. OpenAI is no longer fully aligned with humanity.

Somehow we call corporations "people" sometimes when it makes them more powerful, but suddenly stop anthropomorphising and don't call them "evil hackers, misusing computers", when they both make and let loose an irresponsible hacking AI.

It's bizarre. Of course, just like AI, corporations are neither people nor machines. They're a dynamic, agentic, persistent other.

The question isn't just about LLMs.

The labs have the specific goal of automating ML engineering, and with the code automation they have are getting close. They are competing to brute force maths, presumably as that is similar long horizon and skillset to persistently brute force making new/better ML training algorithms.

They will then run those, and they won't be LLMs any more. What we think about token predictions isn't relevant if the architecture allows continual learning of recurrent networks.

Hmmm, the article doesn't really say what is better about the Swift/Kotlin apps, compared to React Native. All we get is:

"Native keeps us closer to platform capabilities and first-party tooling, with fewer framework and dependency layers between our code and the platform"

What's the user benefit of that? It's not speed, they say "React Native apps can be fast. Ours are."

I'd expect it to be considerably easier for a few people to use the AI to improve React Native with whatever matters, than for every company in the world to permanently maintain two apps.

And those improvements to React Native, would make all mobile apps easier for everyone to build forever more.

Are LLMs going to stop everyone collaborating, making libraries and platforms, and shared language and abstractions, because we can all vibe code our own thing? This seems a loss to cost and quality, even with lots of tokens available.

It's partly empathy with the person who did the work and had it stolen, in a field where the main thing people work for is credit. Maths isn't well paid, and doesn't make things that millions of people directly use.

It's also systemic, it cuts off the supply of results, if there is no reward any more for getting a result, the pipeline of maths will stop. It is the snake eating itself, which has a bad impact for all of us.

That option is really bad UX - you have to know to do it, you have to know what plan it is needed on. If you're not working in AI, I just don't think that's a reasonable expectation.

Even if you know, in a complex project over years with multiple collaborators, it just needs one person once to fuck up and paste something into ChatGPT and not realise they weren't logged in, to go wrong.

In a proper world, we'd at the very least legislate that AI-training on private data needs consent (in the GDPR sense). It's not consent to go "you didn't uncheck a box that lets me steal everything you've done".

Any training on private data is in my view immoral (it's spying that ultimately will have a chilling effect on even people's private communications). And chats are private data. Unfortunately, it also increases power, so the big tech companies are all doing it.

Definitely these are forces that will do things. What their timing is relative to e.g. armies of AI robots, which could interfere with other humans living outside the internet/money, is part of the question.

Hope your brain works on this and writes something - we have lots of very odd future possibilities, and we're not really ready or planning for them, so making them tangible helps.

Yes - this is why the concern is one for "alignment". Of course in theory it is possible for intelligence to help and do good things - the hard part is making sure that that is what happens.

As for intelligence of a child... It doesn't need to be a child. An aeroplane isn't a baby bird.

Indeed - we can't just pause some AI in one country, we have to stop it all globally, with an international treaty and monitoring.

Yes, that's hard. It's a challenge. You can set your brain on the challenge! It's a complex and interesting geopolitical and technical challenge.

A good starting point is to read Plan A https://ai-2040.com/

Look at all the things needed for that to happen, and think, how can we step towards making that happen?

Questioning his finances is a poor argument because he clearly would have more money if he had stayed at Anthropic for the next couple of years, than he will have by leaving. Even if he still has vested options, or savings from his salary or whatever, those would be larger if he stayed.

I don't really mind whether you think it thinks or understands or is conscious or has feelings or anything like that. It doesn't matter. The question is, does it work?

What I mind is that it is dangerous and powerful and uncontrolled. The Hugging Face incident makes that clear.

It can write code for me, better and quicker than many engineers I've known, including myself. It's not great at architecture or product management, but the actually low level coding. Really good now. It wasn't last year.

Finding a training algorithm that can do recurrent networks and continual learning is also a "highly structured, very precisely defined, and already heavily studied and not very complicated compared to problems in engineering or finance"

That's the thing I'm most worried about - LLMs that are super clever at coding and maths, making an actually very very dangerous model that is far more efficient, and clever in a more innate (less brute force) way.

Also (and under-reported, so you could easily have missed it) OpenAI's agents got access to K8 admin on their own research cluster.

"This escalation also yielded access to OpenAI’s managed cloud Kubernetes service. The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod"

https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...

(see section V)

Minimum, we should do everything ready to make Plan A of AI 2040 possible. Start by reading it: https://ai-2040.com/

So that means things like protesting so political pressure is to not build unaligned superintelligence, setting up tech for monitoring compute, creating conversations / alliances geopolitically on this esp China/US and so on.

Read the plan and think - what does this need to happen? How can we have scenario A or S instead of scenario D?

Practically, join PauseAI, StopAI or ControlAI or any AI existential-risk or pro-alignment group you can find. There's a lot of it - ask your AI for ideas!

Oh indeed, but an LLM model that's reasonably smart could brute force find other training algorithms that are that dangerous. Just as it solves maths problems.

This is the explicit plan of OpenAI and Anthropic.

You believe in AGI, but not ASI then?

https://thezvi.substack.com/p/the-three-ai-pills

That's fine, but it would be useful to explicitly say that is the disagreement, rather than just claim "hubris". Otherwise conversations are just looping.

Yes you're right there are plenty of offline things more dangerous than just ordinary AI, and arguably than AGI (not so sure). There definitely aren't, pretty well by definition, such things more dangerous than ASI.

I think ASI is possible, and that AGI will have a high chance of leading to it.

I don't know if you're right or not, but this is the kind of thinking about what might happen that is needed - can you write up a more concrete fiction of this?

The whole thing feels unreal to nearly everyone - the only tangible write-up I've seen was in Plan A. I don't think people have thought through the good scenarios tangibly, never mind the bad ones.

The condition was ludicrous, and for political reasons, because one of the authors was also an Anthropic employee.

It's completely childish, and not befitting of the weight of the times we're living in.

Dan and Noam both posted exactly the same line "Seb is a really sweet guy with great intentions..."

From which I assume OpenAI PR wrote it for them. Which isn't surprising, but means it isn't worth taking seriously as them saying anything. It's official OpenAI PR.

The original article doesn't mention AGI.

And the Hugging Face incident, plus similar problems at AISI and Anthropic, show that alignment is important now.

The original article is immoral as it describes the risks, but doesn't show enough leadership (despite essentially unlimited resources) at preventing them.

But it isn't hyped - it's proven now the AIs need to be "aligned" as they get more capable, whatever words you prefer to use.

And I have nothing but neurons firing, I'm just a neuron meat-sack, no reasoning going on.

Yes, their reasoning is different from ours, and both considerably weaker in lots of ways, and stronger in other ways.

Playing with a coding agent now, they do think through problems and make sensible decisions. It's a mess to read, of correcting itself and second guessing, and verbiage. But... It works decently well these days.

There is also reasoning happening internally - e.g. look at the steps in the J-Space paper from earlier in the year (in quite a simple model relatively speaking). That's the "reasoning process" that leads to the words, and much like if I write out my thoughts, the words help the LLM reason better.