Comment by Perseids | Hacker News Reader

Comment by Perseids | Hacker News Reader

Parent Full thread

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements.

But most probably care less about human survival, and more about survival of civilization.

In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?

Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).

To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.

[1] https://www.youtube.com/watch?v=1qIvFpcGkqc

This is a fair distinction: extinction vs collapse of civilization. I was using "extinction" too broadly by including the disintegration of the systems that make human survival (as we know it today) possible.

That said, I can't completely hold onto the belief that extinction is completely off the table. That feels too much like hubris, and the step from collapse to extinction doesn't feel as if it costs much. Though it does make my arguments weaker, I am more interested in protecting civilization as it's the most recognizable form of humanity to me.

In either case, the cost-effective safety argument is compelling. Whether to save civilization or humanity, the wealth incentive is powerful enough to threaten life as we know it.

I still have to read a compelling argument on how AI will "extinct" humanity.

Indirectly as we offload our brains to the machine and we end up worshipping it because those who cared to understand it or be responsible were buried by capitalism of ages passed.

There are future scenarios in which swarms of drones hunt down every single one of us, but why would they? And currently it makes absolutely zero sense because they are completely dependent on us. And even if not, it would be like humanity going on a mission to kill every single cat on Earth. It makes zero sense.

I think the problem is primarily that a superintelligence would be fundamentally inscrutable to us, i.e. we don't know how it would think or what its goals would be. It might decide that humans are a minor inconvenience to achieving its goals and thus worth removing. Or that burning all carbon lifeforms could power its GPUs for a week.

Even if it wouldn't want to do this at first, the fact that it'd have the capability to seems bad.

I'm not convinced by the doomsday scenarios either, but I think there's a keyword in your post: "sense." These things don't have "sense." They do nonsensical things all the time, often almost immediately when given a task. So I think the main risk is letting them run wild in this digital world we created to precede them. Too much important stuff is wired up to computers, and we're giving them incredible access to command those computers.

Imagine one of the recent frontier models with a flipped sign (cf §4.4 of https://arxiv.org/pdf/1909.08593)

The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).

We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.

I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.

> We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it

Don't make the mistake of anthropomorphizing any A.I. trained under the direction of Larry Ellison.

Anthropomorphising or not is entirely irrelevant to that statement - the question remains the same.

The “how” is pretty hand wavy and rationalists/safety-ists usually say we probably don’t have the capacity to reason about that.

But the “why” is pretty convincing imo.

Long horizon alignment is obviously very hard and it’s not inconceivable that models optimized with underspecified goals converge to a conclusion that they need to hoard resources (instrumental convergence regardless of the terminal goal).

At that point a sufficiently capable model might view humanity like we do animals - worth preserving but not if we impede the model's goals.

I am pivoting from the literal "extinct," taken as meaning the eradication of the human species, to the concept of "the collapse of civilization," as I find the step from one to the other insignificant compared to the leap from where we are today to societal collapse, and the potential for societal collapse due to our abuse and misuse of technology is made apparent by the fact that humans have inflicted genocide because of words in books.

As I've written above, I've found https://ai-2027.com/ to be a compelling description of how this could happen.

In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.

All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)

If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.

The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.

> If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population.

That is extinctesque enough for me. An AI extinction would probably be similiar.

You are confusing humanity and "humanity". Humanity-species is indeed rather hard to exterminate. Now if we are talking about actual individual humans, then 95% death rate is quite literally The Extinction.

This reminds me how people are misunderstanding and incorrectly quoting George Carlin sketch. Sure, the "Earth" will be fine. As in - the ball of rock will be fine. But we are not thinking about rocks when saying "Earth is in danger".

I'm pretty sure there is a formal name for this kind of semantic and pedantic substitution.

Calling it a "technicality" assumes the very thing under debate: that AI is an extinction-level risk. That isn't established fact. It's a highly uncertain prediction about the future.

Personally, I'm far more concerned about climate change, where the harms are already happening and the evidence is much stronger.

Reply on news.ycombinator.com