Hacker News Reader

Comment by DirkH | Hacker News Reader

There was also only "a lot of theory" and zero empirical evidence that nuclear bombs were possible not that long ago. Theoretical fact-based peer reviewed literature often (though not always) precedes empirical evidence.

We shouldn't jump the gun and say runaway AI is inevitable just because there is fact-based theory on its possibility. Just like it would have been dumb to just look at E=mc^2 and conclude nuclear weapon development is 100% inevitable.

But in the same breath we shouldn't say it's impossible just because we don't have empirical evidence either.

View on HN

Prior to the development of nuclear weapons, there was well-grounded peer reviewed theory in the physics literature.

There were predictions that could be made by the theoretical frameworks, and experiments to confirm those predictions.

The literature well described the composition of atoms, the decay of radioactive elements, the concept of fission, and the conversion of mass to energy.

No such groundwork exists to substantiate the claim that runaway AI is possible.

LessWrong, the so-called Alignment Forum, and similar Internet groups propose thought experiments. They couch these thought experiments in the guise of inevitable outcomes.

The prophesies of the influencers originating there have no more relationship to the truth than the average enthusiast’s daydreams.

The content there fills a role similar to science fiction in the atomic age: fanciful speculation on the properties of imaginary systems.

Over time, the bad content is forgotten, and a handful of items survive and look prescient via selection bias.

Don't we likewise have predictions about FLOPs, compute, algorithms etc etc that are grounded in theoretical frameworks, and every year or so we make predictions and make experiments to confirm or deny AI capability predictions?

The AI and neuroscience literature likewise describe the composition of intelligence and what it takes to build it.

What would you need to see to believe we have enough evidence that runaway AI is possible, even if it is highly unlikely any time soon? (Worth adding here I don't know exactly what you mean by runaway AI other than simply AI humanity as a whole loses control over). I'm asking what it would take for you to go from 0% possible to 5% possible.

Last time I checked science fiction authors in the atomic age making fanciful speculation on the properties of imaginary systems weren't responsible for actually building said systems. Though naturally, there will be exceptions. Nonetheless, I still have a hard time dismissing Geoffrey Hinton's concerns as fanciful sci-fi.

Likewise, a nuclear world war actually wouldn't be as bad or even civilization-ending as fanciful sci-fi authors make it out to be. But dismissing its risks entirely because there are some nuclear-war forums where people come up with weird radiation thought experiments just seems preemptive to me.

Reply on news.ycombinator.com