Hacker News Reader

Comment by shrimpx | Hacker News Reader

> ai can make exponentially more powerful ai

Is this true to any extent right now, or just blatantly incorrect assumptions on which to base clickbait and FUD?

The issue is whenever it becomes true it is likely way too late to regulate it.

For example let's imagine a hypothetical world that didn't have guns yet be we think they were possible. Well if you're waiting till one group made guns first you're going to be at their mercy going after their gun armed soldiers with swords.

Right now? Not true to any extent, no.

But it's not exactly "blatantly incorrect assumptions". It's more wishful thinking. (The "self-improving, exponentially more powerful" AI is currently the only hope for an AGI - except that there isn't currently any realistic hope of a self-improving, exponentially more powerful AI.)

Wishful? More like nightmare thinking, hence the desire for some people wanting to ensure that if it can happen, that it doesn't.

I think it's an assumption based on the fact that we don't seem to be anywhere near the maximum amount of computing capacity that we could be using to train AIs. It might be just throwing more compute at the problem, or it might be that we are nowhere close to the ideal algorithms / model architectures, and AI will aid the process of getting there.

To turn it around a bit, what is your null hypothesis? If it's that AI will not be able to make more powerful AI, why do you believe that?

Given that there are two cases, and smart people disagree about which one we're headed towards, and one could have very dire consequences while the other is no big deal, I think it's reasonable not to assume that the latter is true without exercising a lot of due diligence.

"We can apply more compute power" is something very different from "exponentially self improving". Assuming that "exponentially improving" will show up if given enough compute power is very much not a null hypothesis.

Ok, so there's self improving, where an AI tries to make itself smarter, and then there's just exponential improvement, where an AI does one or more of the following, with or without a human in the loop:

* acquires money through legitimate business and buys more computing power to train a smarter AI.

* acquires money through illegal means and does the same.

* designs a more efficient model architecture or training process that results in a smarter AI.

I don't know how far away from this we are, it's very hard to predict. I do predict that at first we'll see mostly things where a human is directing the AI, but there's selective pressure toward not having a human involved, because someone who just sets up an AI and lets it loose is going to have a thing that reacts faster and can work 24/7. Irresponsible businesses that let their AIs run rampant are going to make more money, and be able to afford better AIs. It's the same story that has played out again and again over history, Moloch (1) pushing us to do the selfish but harmful thing by rewarding those that do it.

Remember that exponential curves seem pretty slow at first, and it can be hard to tell when you are standing on one. It might be that we're still years away from exponential self-improvement, but I think it's worth taking seriously as a possibility. Even with just humans designing AIs, the amount of effort required has become exponentially less as we got more compute, designed better libraries and tools, people published research and copied it and built on it. We can't predict exactly what or when the next step will be, but I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic.

[1] <https://slatestarcodex.com/2014/07/30/meditations-on-moloch/>

> I don't know how far away from this we are, it's very hard to predict.

On this point, I am in total agreement with you.

One way to tell might be, what happens if we let an AI try to improve itself? Does it get better, or worse? My money currently is on "worse". Well, how much worse? How far away are we from it being able to improve itself? I think we are an enormous distance away, almost an immeasurable distance, but I could be wrong.

> I think it's just as reasonable to assume that the curve of this progress is exponential as it is that it's logistic.

It's always logistic in the end. (But if we're on the first part of the curve, where it's exponential, that "in the end" bit doesn't matter.)

I believe AI could make exponentially better AI but do we have any evidence that this is likely, or could happen soon? Sounding alarm bells with no understanding of how likely it is seems irresponsible.

Reply on news.ycombinator.com