Comment by josephg | Hacker News Reader

Comment by josephg | Hacker News Reader

Parent Full thread

Imagine a bag filled with marbles. We can't see inside the bag, we can just pull marbles out of the bag. Some marbles are white - they improve the world in some way. Wifi. Trains. Electricity. Some marbles are grey - they are useful but dangerous. Firearms. Nuclear weapons. Gene editing. And some marbles are black. If we pull a black marble out of the bag, we hurt or destroy ourselves.

Humanity's strategy so far has been to pull marbles out of the bag as fast as we can. And that's been working out, because most marbles are white. But we're insanely lucky we haven't had a nuclear war. The USSR's early warning systems picked up 5 incoming american missiles in 1983. They didn't fire back because one of the officers on shift didn't believe the reports (why only 5 missiles?), and he declared the reports were fake.[1] If a different person was on shift, there's a good chance nuclear war would have broken out because of a sensor glitch. And that's far from the only instance. The list is terrifying: [2]

And now we're improving AI capabilities as fast as we can, despite it being pretty clearly another grey ball. AI has a lot of amazing uses. But why are we rushing so much? Doesn't seem wise.

[1] https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar...

[2] https://en.wikipedia.org/wiki/Nuclear_close_calls

> Humanity's strategy so far has been to pull marbles out of the bag as fast as we can.

> But why are we rushing so much?

Talking about these things with elaborate analogies only obscuring the point you already stumbled into: Humanity advances because humans are curious, driven, like to discover things, and like to push the boundaries.

Talking about humanity like one entity that operates together misses the point: There are billions of people in the world and some of them are going to discover new things.

There is no general counsel where humanity gets together and decides what humanity works on. There is no singular bag with a singular “humanity” pulling marbles out one at a time.

The analogy is too broken to be useful for the discussion.

There’s a point at which logic/reason/wisdom overrides our curiosity, and there are quite literally multi-nation groups that collaborate on matters of global importance.

There are plenty of examples of this, e.g. when we discovered how to harness nuclear energy, we quickly realized that extreme measures needed to be taken to ensure we don’t collectively destroy ourselves.

> There are plenty of examples of this, e.g. when we discovered how to harness nuclear energy, we quickly realized that extreme measures needed to be taken to ensure we don’t collectively destroy ourselves.

You’re speaking of “we” as if the whole world was operating in unison, while there’s a war going on in Iran over nuclear disagreements.

The nuclear situation only reached stability after multiple countries fully developed it. I don’t see the parallels to halting AI.

It's hard for me to understand how the parallels wouldn't be abundantly clear.

The nuclear stabilization that occurred was the result of a collective understanding of the existential risk it posed leading to anti-proliferation agreements and superpowers going to great lengths to prevent nations deemed as unstable from obtaining such technology.

Some portion of the global community may collectively discover/decide that the risk AI presents is worthy of anti-proliferation treaties and wars may be fought over it.

The main gap in the comparison is that we have a much clearer understanding of the risks of unchecked nuclear weapons use than we do about what current and future AI tech might or might not unleash.

The underlying point is that we have historically placed limits on our curiosity when we deem certain paths of exploration existentially dangerous. It has yet to be seen if AI will reach those levels of existential risk, but if they do, there is plenty of precedent for stepping on the brakes.

> There is no general counsel where humanity gets together and decides what humanity works on.

Nuclear Non-Proliferation Treaty (1968)

Biological Weapons Convention (1972)

Asilomar Conference on Recombinant DNA (1975)

We've done this sort of thing plenty of times around specific, dangerous technologies.

And countries still violate these treaties. See Iran and NK.

Also the bio/DNA ones are quickly falling apart as the technology to do these kinds of things on a small scale are readily available. To the point where we're finding illegal wet labs in the US.

Nuclear was more of an exception because of many of difficulties in purifying stuff for weapons and its detectability.

AI is a much, much, much harder problem is it uses the already existing infrastructure as a base. Unless we regulate, monitor, and ban all private usage of technology AI can be developed at a small scale.

Is your argument essentially that because global treaties haven't been universally effective in all cases, we should just YOLO AI even if it has the potential to cause great harm?

No, we shouldn't yolo AI, but because we started the ball rolling and it may be impossible to stop.

> Humanity advances because humans are curious, driven, like to discover things, and like to push the boundaries.

This is true, but not all the time, and transparently not the case for the current leaders in AI development. Their motivator is money.

Whats wrong with gene editing?

Reply on news.ycombinator.com