Comment by manish_gill | Hacker News Reader

Comment by manish_gill | Hacker News Reader

Parent Full thread

Except, in the case of climate research, we have a plethora of evidence of the possible harmful effects and we can see it happening today. Everything about the potential harmful effects of AI is pure conjecture, because there is no human-level general intelligence AI system that exists today. It's, once again, something philosophers like Bostrom will make a career out of.

I'm 100% with Torvalds on this when he laughs at the prepostorous notion that AI will become a doomsday scenario. I think it'll become more and more specialised, branch out to other fields and become reasonably good. But there's a huge leap to go from there to HGI.

> any powerful system that optimises for something that's different from what humans want.

Except this notion rests on the premise that humans will not be in full control, which leads to the exponential growth argument which leads back to the Skynet like scenario.

> the point is that even uncertainty is enough for action

And that action is...what exactly? People won't stop building intelligent systems. There is no real path that we have from where we are to HGI, so it's not like researchers have a concrete path. Just what exactly does this research look like?

> "We've got no idea what the chance of run-away climate change is, so we shouldn't do anything about it."

Extremely Poor analogy. We have decades worth of concrete data that tells us the nature and reality of climate change. We demonstrably know that it's a threat. Can you say the same about AI?

Also, I'll take the time to re-iterate how heavily skeptical I remain of groups like MIRI that are spearheaded by people who don't believe in the scientific method, believe in stuff like cryogenics, have history of trying to profit off of someone else's copyrighted material and have someone managed to convince a whole lot of people that donating them is the best way to fight off the AI doomsday scenario. People should do their research before linking to stuff like that. :(

I'm not saying we don't know whether climate change poses a tail risk or not (it obviously does). I'm just saying that claiming uncertainty isn't a good reason to avoid action.

In general, if there's a poorly understood but potentially very bad risk, then (a) more research to understand the risk is really high priority (b) if that research doesn't rule out the really bad scenario, we should try to do something to prevent it.

With AI, unfortunately waiting until the evidence that it's harmful is well established is not possible, because then it could be too late.

What AI risk research could involve is laid out in detail in the link.

"poorly understood but potentially very bad risk" is something you could say about the risk of an alien invasion.

> And that action is...what exactly?

I would guess A. the development of "known safe" high-level primitives, and B. a coupling of education in the craft of AI with education in the engineering of AI.

The "profession" of developing strong, general-purpose AI should basically look like a cross between cryptography and regular old capital-e-Engineering: like crypto, you'd constantly hear important things about "not rolling your own" low-level AI algorithms; and, like Engineering, the point of the job would be making sure the thing you're building is constructed so that it doesn't "exceed tolerances."

The research goals in the field of "friendly AI" are thus twofold:

• work in understanding possible computational models for minds, to understand what sort of tolerances there can be—what knobs and dials and initial conditions each type of mind comes with;

• and work in development of safe high-level primitives for constructing minds.

Both of these are the subjects of active papers. Note that these can both also be described as plain-old "AI research"; they're not just abstract philosophy, these papers are steps toward something people can build. The research within the subfield of "friendly" AI just has different criteria for what makes for a "promising avenue of research", e.g. ignoring black-box solutions because there are no knobs for humans to tweak.

> MIRI ... spearheaded by

MIRI is two things, and it's best to keep them mentally separate. It's a research organization, like Bell Labs—and it's a nonprofit foundation that funnels money into that research organization.

Yudkowsky is the director (head cheerleader) of the nonprofit. He doesn't really touch the research organization. The research org will succeed or fail on its merits (mostly whether it hires good researchers), but the leadership of the nonprofit has not-much to do with that success or failure, any more than AT&T had any impact on the success or failure of Bell Labs. You can believe in the research org known as MIRI even if you actively distrust Yudkowsky.

"have history of trying to profit off of someone else's copyrighted material"

That's quite a strong claim - could you provide evidence for it? Are you referring to HPMOR? HPMOR has always been given away for free, both online and in print, and J.K. Rowling has explicitly allowed non-commercial Harry Potter fanfics.

He was eventually dissuaded because a vocal majority of HP fans were against it (because JKR has only allowed non-commercial use). But here is the original announcement I found from a forum -

The Singularity Institute for Artificial Intelligence, the nonprofit I work at, is currently running a Summer Challenge to tide us over until the Singularity Summit in October (Oct 15-16 in New York, ticket prices go up by $100 after September starts). The Summer Challenge grant will double up to $125,000 in donations, ends at the end of August, and is currently up to only $39,000 which is somewhat worrying. I hadn't meant to do anything like this, but:

I will release completed chapters at a pace of one every 6 days, or one every 5 days after the SIAI's Summer Challenge reaches $50,000, or one every 4 days after the Summer Challenge reaches $75,000, or one every 3 days if the Summer Challenge is completed. Remember, the Summer Challenge has until the end of August, after that the pace will be set. (Just some slight encouragement for donors reading this fic to get around to donating sooner rather than later.) A link to the Challenge and the Summit can be found in the profile page, or Google "summer singularity challenge" and "Singularity Summit" respectively.

Reply on news.ycombinator.com