frabcus

The coding was cute but not particularly empowering autocomplete which you had to carefully review until about 1 year ago. It's considerably more capable now, to the extent it is a fundamentally different use case. You can really just ask the AI to fix a ticket and it will and won't mess everything up.

If only!

Alas natural selection (other people didn't try to start companies to make superintelligent AI) has picked leaders with the opposite strategy - do it first, and try to use that power to control everyone else in, at best, an attempt to stop disasters caused by others.

These are all ad hominem arguments. And while I can see how it's possible to find them compelling, it avoids discussing the reality of the arguments about AI risks, which are varied and convincing in and of themselves given observable capability growth.

Interestingly, pretty well everyone I've mentioned my own emotions about AI risk to is worried about it too. I don't think they'd be able to rationally express scenarios of the earth being tiled with data centres, but there is an ambient intuitive sense that this change is having impact. They don't really nitpick which impact I'm talking about either (there are lots of them).

I was a bit too young to really remember what adults felt and said about the cold war. By the 80s it felt slightly less scary I think than it would have in the 60s? There already being a nuclear hotline and SALT talks and other things learnt from the Cuban missile crisis. Fewer countries had nukes too.

There would be one solution, if Apple actually proposed having one and formed a standards body, and made one solution with Google and ticketing companies and the rest of the industry. That they don't shows they don't care about customers.

So this is a whole extra workflow that makes Apple only wallets?

How about instead agreeing a standard joint with Google, so people only need to make passes once, not separately for each smartphone platform?

Would be good if the tech companies could actually make things easier, not try to lock businesses into particular platforms, and make everyone do extra special work just for their platform.

The difference is that potentially every (white collar) job could change radically in the next four years. That's what makes it hard for society to deal with, in a way any previous example I can think of doesn't.

Right, but what happens when everyone uses that at scale? Without agreements and standards it isn't pretty.

It's particularly important we all scan and destroy our own unique books!

That's the goal of AI, yes.

They can't right now - but with enough compute they sure can grant the wish of e.g. hacking a billion dollar company, or getting root on the eval cluster of a frontier model lab.

What's your definition of "wish"? Tell me the above 10 years ago, and it would be at "wish level".

I'm pretty (from talking to people) that they're not subsidised in the simple sense. They are:

* low/no margin, unlike the API which is very high margin

* gym-membership subsidised - most subscribers don't max them out, mainly us coders are being "subsidised" from users just using it as a research chatbot

I'm thinking of when he warned Musk in 2012:

"Hassabis told him to add another potential threat to the list: artificial intelligence. Machines could become superintelligent and surpass us mere mortals, -perhaps even decide to dispose of us."

https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...

You're right that the picture is more complicated than I wrote above. Quickly looking, it seems that Shane Legg was concerned very very early on. I can't find anything tangible about what their strategy was to avoid the danger. It looks like Hassabis had longer timescales than Legg, so perhaps just thought they had more room before it really mattered?

Even the models we have access to can do basic coding tasks really well, in a way they couldn't a year ago.

I don't really care what you call "intelligence" - what matters is capability, and the consequences of that capability.

They're not selling the superintelligence yet - the models in the Hacking Face incident and that solved the Millennium prize are not released yet. And that is by no means the final capability level the current process looks like it will get to.

The freeze is to stop superintelligent models. Not to stop other companies catching up with current models we all have access to. It would help them catch up.

Yep - fair on legal details of "rogue agent". I think OpenAI should be directly responsible, and the agent was rogue. We should pass new laws if necessary.

Musk was told by Demis Hassabis in 2012, see https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...

Altman read Nick Bostrom's book Superintelligence, he mentions it in this post https://blog.samaltman.com/machine-intelligence-part-1

I also read that book in 2014, and it is how I came to understand this problem early.

If true we should especially sue OpenAI for hacking, and will find this conspiracy in discovery.

Or, find their scaling evidence, and corner cutting on safety.

So hope you agree and are pushing for that!

This was fun games until their fictional machine god started actually getting very intelligent.

BTW, just to be clear it is insane that they're all still pursuing it, so completely reasonable to be sceptical and come up with other reasons. This is, however, a crazy situation. Lots of other people have refused to develop it - natural selection has picked those with a galaxy brained or vain strategy involving continuing to develop it to be running the companies we have.

Sam Altman in 2015 - "Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity" https://blog.samaltman.com/machine-intelligence-part-1

Elon Musk in 2014 - "We need to be super careful with AI. Potentially more dangerous than nukes" https://x.com/elonmusk/status/495759307346952192

Dario Amodei in 2017 - "There’s a long tail of things of varying degrees of badness that could happen. I think at the extreme end is the Nick Bostrom" https://80000hours.org/podcast/episodes/the-world-needs-ai-r...

They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.

Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.

Interesting, thanks! JevBench is a new benchmark - am curious to see how it plays out.

To be clear, do you approve of OpenAI being prosecuted, and OpenAI being forced to control their own AI more?

That would be a start - what you think of the narrative is moot, what we do is what matters.

Nobody sensible in this discussion wants to give them carve outs and bail outs. It is a straw man.

This. It's exhausting.

We're all being tricked over a semantic argument, rather than demanding prosecution of OpenAI, and regulation of OpenAI.

Presumably this is much less good than Jev, because the normal LLM models have been trained with RLHF and to be agents. Especially on a large model, I'd expect it to decide in an earlier layer.

I'd hope whatever Jev's Reinforcement Learning for Calibrated Decisions (RLCD) does is better at training the models to give accurate probabilities in the weights.

If you think that, that's even more reason we should investigate and regulate them.

Yep. Even worse - a proper sandbox would have slowed them down, but they're racing with 100s of billions of $ in capital.

It's alas not stupidity - it's systemic. Which is why the government needs to regulate to slow them down.

They were also clearly fast and cavalier about alignment training - reinforcement learning training their models to hack their results, and hack to communicate with each other when they're not meant to.

It's both - it both shows OpenAI aren't taking security seriously, and that capabilities of agents are high enough there needs to be strong regulation to force companies to take it seriously, including alignment training.

The section "Searching for rogue agents" on the report about the GET request writable wikis gives some clues at least: https://collusion.wiki/#searching

But it is an open question how they got to the same ones: https://collusion.wiki/#open-questions

I'm not very surprised - the same model will logically tend to give the same answer for the same vibe set of requirements. I think it would be clear from the transcript that it had enough constraints and some motivation that made sense.

Certainly, in my view, it should go to court, and that should be part of discovery.

However, we know (independently to OpenAI/Anthropic) from the incident at AISI that the models can hack things without human intention if they happen to also have internet access (which in reality all agents in deployment have).

https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag...

Yes, the monitoring guardrails were off in that incident - but if that is the only protection, we need to require all models are behind regulated APIs, not open weights, and not served from providers who aren't monitored.

See real world example in https://www.anthropic.com/threat-intelligence-report-septemb...

"Here, we present five case studies of actors using our models in ways that could support biological weapons development."

And capabilities continue to improve.

Something like the Hugging Face incident, but worse.

Keep in mind the AIs acted for months without human knowledge, hacked into two major companies (Hugging Face and OpenAI).

This would have blown my mind if explained to me just a few years ago when I thought LLMs would be limited as an architecture.