Comment by WhyOhWhyQ | Hacker News Reader

Comment by WhyOhWhyQ | Hacker News Reader

Parent Full thread

Robert Wright just posted a (somewhat) interesting conversation with one of the authors.

His thesis involves at least two ideas (1) projects which could exponentially increase our AI capability are just around the corner (will happen by the end of this year or some time next year at the latest) (2) it's possible for state actors to deter those projects with sabotage (he coins the term Mutually Assured AI Malfunction).

It doesn't make sense to me however because the cost of the next AI breakthrough just doesn't sound comparable to the cost of creating nuclear weapons. With nuclear weapons you need this extremely expensive and time consuming process, and you need to invest in training these extremely skilled people. With AI, the way everyone seems to talk about it, it sounds like some random undergraduate is going to come along and cause a massive breakthrough. We've already seen Deepseek come along and do just as well as the best American companies for practically pennies on the dollar.

Also; it is obvious when someone uses a nuke. There is a big crater and a mushroom cloud + lots of radiation. It isn't half as obvious that someone is using an AI, particularly once they start to obscure it. If a military campaign is executed with apparently superhuman efficiency, does that mean AI was involved or just that the people involved were good? There'll always be plausible deniability if it matters.

People underestimate just how bad human management is; we haven't had an improvement on it to date apart from some mathematical techniques but even just getting the basics right consistently would probably give an army a big advantage if they work anything like a more standard corporation. Which they will; there are no magic techniques to be more capable when guns are involved. A superintelligence could probably win just by being demanding about getting basic questions answered like "Is there a strategic objective here? Is it advantageous to my side if that objective is achieved? Can it reasonably be achieved with the capabilities I have?" and not acting when the answer is no. That'd put it ahead of the military operations the US has been involved in this century. Bam, military superintelligence with plausible deniability.

> People underestimate just how bad human management is; we haven't had an improvement on it to date apart from some mathematical techniques but even just getting the basics right consistently would probably give an army a big advantage if they work anything like a more standard corporation.

Don't overestimate the efficiency of civil big organisations (what you call "standard corporation[s]") - they have the same kind of problems.

First you have to get officers willing to follow an AI.

Did he go into what those ideas are in particular? Modern AI has 2 big shortcomings compared to humans right now imo: humans learn MUCH faster and humans are a lot better at solving novel problems. If they can make progress on this, I'd wager human intelligence is in danger.

In the podcast the example he gave as a kind of project which could lead to "super-intelligence" (the kind of project nation states would be interested in sabotaging, according to his theory) is one in which the AI is trained to be an LLM researcher. The idea then is to run thousands of these LLM researchers to train the next generation of LLMs. Their paper probably has more specifics.

I don't remember exactly what was suggested would be the forms of sabotage. He also suggested export controls on the high value chips used for training and running models.

Reply on news.ycombinator.com