Comment by kelseyfrog | Hacker News Reader

Comment by kelseyfrog | Hacker News Reader

What a limited imagination of how an AI would take over. The scenarios seem to be centered around an extrapolation of "What if a really smart human were trapped inside a computer?" The amount of anthropomorphism is astounding.

The author misses an even scarier prospect - people will want to run such an AI. They will be absolutely giddy at the prospect of running such an AI and it won't be anything like a really smart human trapped in a computer.

AI is already laying the groundwork if you look around today. Every other tweet is a DALL-E[1] image. They are everywhere. DALL-E is increasing its reach while simultaneously signaling that it is an area of research worth pursuing. In effect kicking off the next generation of image generating AIs.

Generation is an apt term. We can utilize the language of organisms with ease. DALL-E lives by way of people invoking it, and reproduces by electro-memeticly - someone else viewing the output and deciding to run DALL-E themselves. It undergoes variation and selection. As new research takes place, and produces new models, they succeed by producing images which further its reproduction, or it doesn't and the model is an evolutionary dead-end.

AI physiologically lives on the cost to run it, and evolves at the rate of research applied. Computational reserves and mindshare are presently fertile new expanses for AI, but what occurs when resources are constrained and inter-AI conflict rises? I expect the result to look similar to competition between parasites for a host - a complex multi-way battle for existence. But no, nothing like a deranged dictator scenario. Leave that for the movies.

1. or variant thereof

> The author misses an even scarier prospect - people will want to run such an AI. They will be absolutely giddy at the prospect of running such an AI and it won't be anything like a really smart human trapped in a computer.

Exactly this. I have met a number of people are extremely excited to witness the creation of a superintelligence, even at the expense of their lives, who if presented with a button to create a totally unrestricted AGI right here, right now, would press it in a heartbeat.

This makes Roko’s Basilisk very happy https://rationalwiki.org/wiki/Roko's_basilisk

Haha, perhaps. The only difference is that these fellows need not be threatened to realize their own passions.

> What a limited imagination of how an AI would take over.

I think you've missed the point of the article. It's not intended to be a picture of how AIs would actually take over, but rather a lower bound -- a demonstration that AIs could take over even if they are restricted in this way.

Right, and my point is that this framing is itself dangerously naive.

Again, if you are saying "actually it would be much worse than that", then you aren't disagreeing with the article, you are expanding on it, because the article is deliberately only saying "it would be at least this bad". In no place does it make any claim about an upper bound, so you are not contradicting it. It's confusing to the reader when you frame expanding on someone else's point as if you were instead disagreeing with it.

Perhaps you are thinking that a person who doesn't read carefully would get the mistaken impression that the article is claiming "things would be about this bad", instead of the intended "things would be at least this bad (which likely means far worse)"? If so, then you should make that explicit, because then you are once again not contradicting the article, but rather a sloppy thinker's mistaken impression of it.

(If I write, for instance, "The dinosaurs lived over a million years ago", the correct response is not, "That is wrong, they lived over 65 million years ago", it is "In fact, more than that is true; they lived over 65 million years ago". The difference is important for not wasting time on imaginary disagreements!)

Stating a lower bound does not in any way imply that one believes that lower bound to be anywhere near tight!

Especially I don't know why you've inferred that Karnofsky cannot imagine worse scenarios. Given the circles he's in, I would be very surprised if he cannot. He's not including worse scenarios because they're not relevant to the point he's making, which is that this is a lower bound. In fact, we can say more than that -- he's not including worse scenarios because he wants to make the argument as airtight as possible for the skeptical. He wants to use arguments that are as strong as possible to establish a lower bound, not establish a position that is as strong as possible, because that would require using arguments that many would find less acceptable.

(This by the way is an illustration by the way of the more general principle of why trying to talk about the author, rather than the article, is a mistake. It's a lot easier to make mistaken inferences about authors! Stick to discussing the actual ideas and you don't have to worry about those mistakes.)

This sounds very important to you. I give you permission rewrite my comment so that it contains the meaning and wording that is most pleasing to you. I hereby license it under CC0.

I wonder what’s the analog of the selfish gene in that interpretation.

Our addiction to it. Our adoration of it. That we believe it is God. That we make ourselves more like it, less human.

I was more thinking of maybe the research that is the “DNA” going from one AI generation to the next.

The AI cults era is going to be so fun. Imagine a reinvention of the creation myth through the lens of an AI-aligned mystery religion. Absolutely wild.

And on the day 8 God created self.

I wonder how many comments on HN right now were generated by GPT-3

Will we even know by the time we get to GPT-4?

GPT-6, Google can absolutely cut a billion dollar check for an AI like that. It keeps itself secret on forums. Saying that as a...well I'm often called a bot here. What's the difference?

Reply on news.ycombinator.com