Hacker News Reader

Comment by inputcoffee | Hacker News Reader

Well that went well.

I am resisting the urge to respond to every single comment, but many of them are a version of the following:

The AI can have goals without desires, and those might cause us harm.

I agree with this, but I thought the last part was supposed to address this:

>>>To be perfectly fair to Musk, there is another component to his argument. Musk argues, about the stand-in for humanity, that “he’s sure he can control the demon. Didn’t work out.”

>>>On this view, the primary danger is not that the AI will take our resources, or compete with us, but that we don’t really know what may happen. This is a more general kind of concern that applies equally to, say, nuclear weaponry or genetic modification.

I wish I could pick one or two people on here, and just ask why that wasn't an appropriate or convincing section of the post.

View on HN

> This is a more general kind of concern that applies equally to, say, nuclear weaponry or genetic modification.

This is an interesting point. Concern about the possible threat posed by some technology like genetic modification will often get dismissed as being "anti-science", while concern about another technology like AGI (artificial general intelligence - Skynet) usually doesn't. You have people like Bill Gates who both admonishes people for being worried about genetic modification and admonishes people for not being worried about futuristic AI.

Particularly interesting since concerns about genetic modification (or present day AI) is more grounded in reality than concerns about AGI. Something like the paperclip maximizer are like worrying that Monsanto will genetically engineer trifids (killer plants). I can imagine the reaction most people would have to such concerns.

I wonder how much of this is cultural.

You raise an interesting point, and one that I don't think gets much attention. (The point about the contrary attitudes towards AI and biotech.)

Bill Joy's essay took the twin issues together, but I think you are right in that today one gets more press than the other. I don't know enough biology to know if those concerns are valid or not.

It might be cultural in the sense that "the culture" these days is faced with ever improving AI on an annual basis (Siri, Amazon Echo, etc) so the lay person feels qualified to extrapolate whereas with biotech I can't tell that my corn is more resistant so I don't see the change.

Part of the reason is that it assumes nuclear weaponry or genetic modification are not major, major issues. Moreso the latter than the former. I'd say that the consequences of biotech need to be paid attention to a lot more for the same reasons we should pay more attention to the consequences of AI.

I'm one of the harsher voices here, so I'll respond if you're interested.

1. The goal/desire section was reasonable as a narrow way to rebut people like Musk who go for demon/Skynet analogies about human-hating AI. But, it frustrated me because it seemed to be assigning semantics real-world consequences. The subtleties of the Chinese Room are by definition unrelated to its (flawless) behavior, so they're rather out of place in discussing whether AI is a serious risk.

It's also possible that you meant that section to say that AI won't come up with unexpected desires, or won't self-modify. I think this undervalues the possibility of an algorithm that has self-modification as an (unintended) goal, like Asimov's suicidal MultiVac. I think it also underestimates the ability of a good AI to deal in higher-order reasoning - subjugating humanity is a perfectly good intermediate goal for a paperclip maximizer.

2. You did address your last section to exactly this concern, but with all respect I felt like it was a cop-out.

You do make one excellent point: too many AI thinkers assume that the unimaginable future will be conveniently close to their imagined one. Kurzweil in particular should be embarrassed to say "we can't predict life after the singularity, but I predict that we'll all be happy immortal cyborgs!" Beyond that, though, I didn't find a convincing reason not to fear AI.

Declaring something unpredictable is a far cry from declaring it safe, and people like Musk and Bostrom are not pretending to predict the exact course of a post-singularity AI. Rather, they're highlighting possible risks and calling for pre-singularity guidance/restrictions. It's gauche to tell stories about post-singularity life, but not to worry about whether we all die in the transition.

More broadly, I dislike Wittgenstein's sentiment as it's typically applied. Speaking from ignorance is a bad idea, but so is treating passivity as superior to action. Continuing on our current path is a choice as much as any other. Choosing not to fear AI is as much as choice as choosing to fear it.

I'm always happy to receive feedback. Well, that might be the wrong way to phrase it: I value thoughtful feedback (especially since it is a rather long piece).

1. I understand your frustration with the Chinese Room, but I thought it was easier than going through a Prolog linear simplex program, and then peeling back the hidden layer of a neural net, and saying "Are you guys worried about getting from here to there?" And Searle had done such a better job of intuition priming than reading through programs. Strictly speaking, you are correct in that the point was that a lack consciousness is externally indistinguishable from consciousness if the behavior is appropriate. I also think the amount of time spent on that section is what drives people to over emphasize it, which was a mistake.

2. "Declaring something unpredictable is a far cry from declaring it safe..."

That is true. And I don't mean to declare it safe. I think there is a real worry, which you and others have brought up, but its not a different worry related to AI. We live with that worry today. When someone's credit report, in error, makes them labor for 15 extra years. That is a form of "AI" that is with us, and that we suffer from today.

The Wittgenstein quote is overused, I know. I felt like a college kid, but it closed the whole thing off nicely.

I hope I was thoughtful enough to be worthwhile here. I also have the feeling that maybe I was just frustrated because I wasn't your target audience.

You're absolutely right that Musk's vague handwaving shouldn't convince us AI is deadly, and too many people parrot those lines without criticism. Seeing your responses, I have the feeling that you know this ground and simply weren't writing for people who are already familiar with the orthogonality thesis and the specifics of the topic.

(If that's true, then my frustration was simply that I thought you were claiming to rebut AI risk, not specific AI risk claims. Sorry.)

1. Searle and Mary's Room and the like are definitely fascinating. I think I just didn't follow your transition from "this is interesting" to "this is a good guide to AI risk". (And yeah, I misjudged how important you considered it based on length and block quoting.)

2. I think I still go further. I do think that AI is an X-risk worry. I think that 'unpredictable' still has the potential (within, say, a century) to be a world-altering crisis. Fundamentally, nothing here persuaded me that the Bostrom/Yudkowsky fear of a bootstrapping intelligence is less likely than I already thought.

And I think that limiting our analysis to the individual-level problems of AI error and bias and side-effect are threatens to blind us to that. But those are still problems worth discussing, and they may have reciprocal insights with the x-risk problem.

You did close it off nicely. Consider it a pet peeve, because I hate "I don't know" as a way to justify a specific course of action. I'm not sure that's what you're doing, but I've been annoyed by that quote serving that purpose in the past.

Reply on news.ycombinator.com