Superintelligence: The Idea That Eats Smart People
idlewords.com
idlewords.com
The real AI risk isn't an all-powerful savant which misinterprets a command to "make everyone on Earth happy" and destroys the Earth. It's a military AI that correctly interprets a command to kill a particular group of people, so effectively that its masters start thinking about the next group, and the next. It's smart factories that create a vast chasm between a new, tiny Hyperclass and the destitute masses... AI is hardly the only technology powerful enough to turn dangerous people into existential threats. We already have nuclear weapons, which like almost everything else are always getting cheaper to produce. Income inequality is already rising at a breathtaking pace. The internet has given birth to history's most powerful surveillance system and tools of propaganda.
The only thing which keeps you alive is the unwillingness of your neighbors and those who surround you to kill you. The law might punish them afterward, but extensive research has shown that it provides no disuasion to people who are actually willing to kill someone.
A military AI being used to wipe out large numbers of people is exactly as 'inevitable' as the weapons we already have being used to wipe out large numbers of people. The exact same people will be making the decisions and setting the goals. In that scenario, the AI is nothing but a fancy new gun, and I don't see any reason to think it would be used differently in most cases. With drones we have seen the CIA, a civilian intelligence agency, waging war on other nations without any legal basis, but that's primarily a political issue and the fact that it can be done in pure cowardice, without risking the life of those pulling the trigger, which I think is a distinct problem from AI.
To be fair, it's a small step from effective AI that doesn't malfunction, to an AI over which humans have lost control. It's precisely one vaguely specified command away in fact, and humans are quite excellent at being imprecise.
Anyone power that has significant leverage over another power already has that ability.
A bunch of 'super smart evil AI robots' will not be able to physical deter/control 500 million Europeans - but - a small Army of them would be enough to control the powers that be, and from there on in it trickles down.
Much the same way the Soviets controlled Poland et. al. with only small installations. The 'legitimate threat of violent domination' is all that is needed.
So - many countries already have the power to do those things to many, many others via conventional weapons and highly trained soldiers. That risk is already there. Think about it: a decent soldier today is already pretty much a 'better weapon' than AI will be for a very, very long time. And it' not that hard to make decent soldiers.
The risk for 'evil AI robots' is that a non-state, inauthentic actor - like a terrorist group, militia etc. gets control of enough of them to do project power.
The other risk I think, is that given the lack of bloodshed, states may employ them without fear of political repercussions at home. We see this with drones. If Obama had to do a 'seal team 6' for every drone strike, many, many of those guys would have died, and people coming home on body bags wears on the population. Eventually the war-fever fades and they want out.
Whatever goals the AI has, it will certainly be better at achieving them if it can stay alive. And it will be more likely to stay alive if there are no humans around to interfere. Now you might say, why don't we just hardcode in a goal to the AI like "solve aging, and also don't hurt anyone"? And ensure that the AI's method of achieving its goals won't have terrible unintended consequences? Oh, and the AI's goals can't change? This is called the AI control problem, and nobody's been able to solve it yet. It's hard to come up with good goals for the AI. It's hard to translate those goals into math. It's hard to prevent the AI from misinterpreting or modifying its own goals. It's hard to work on AI safety when you don't know what the first strong AI will look like. It's hard to prove with 99.999% certainty that your safety measures will work when you can't test them.
Things will not turn out okay if the first organization to develop strong AI is not extremely concerned about AI risk, because the default is to get AI control wrong, the same way the default is for planets to not support life.
My counterpoint to the risks of more limited AI is that limited AI doesn't sound as scary when you rename it statistical software, and probably won't have effects much larger in magnitude than the effects of all other kinds of technology combined. Limited AI already does make militaries more effective, but most of the problem comes from the fact that these militaries exist, not from the AI. It's hard for me to imagine an AI carrying out a military operation without much human intervention that wouldn't pose a control problem.
--------- Edited in response to comment--------
These are not bad arguments you are making, or hard ones to get behind. There are just added layers of complexity that the author would like us to think about. Things like how we could actually 'hard-code' a limit or a governor on certain types of motivation. Or what 'motivation' is even driven by at all.
I think you'll enjoy the originally linked article. It's got a lot to consider.
This is a sequence of deductive reasoning that you brought up there. Quite natural for human beings, but why would the paperclip maximiser be equipped with it?
Seriously, the talks specifically argues most of the points that you brought up.
Shit is complicated yo. Complicated like the world is - not complicated like an algorithm is. Those are entirely different dimensions of complicated that are in fact incomparable.
This scenario I agree with. For instance: the AI decides that it doesn't want to live on this planet and consumes our star for energy, or exploits our natural resources leaving us with none.
The whole AI war scenario is highly unlikely. As per the article, the opponents of AI are all regarded as prime examples of human intelligence - many of them have voiced opposition to war and poverty (by virtue of being philanthropists). Surely something more intelligent than humans would be even less inclined to wage war. Furthermore, every argument against AI posits that humans are far more important than they really are. How much time of day do you spend thinking about bacteria in the Marianas Trench?
> AI control
My argument is with exception to this scenario. By attaching human constraints to AI, you are intrinsically attaching human ideologies to it. This may limit the reach of the superintelligence - which means that we create a machine that is better at participating in human-level intelligence than humans are. Put simply, we'd plausibly create an AI rendition of Trump.
Why would this ever apply? We're building them, not picking them out of a hat.
You know, you don't need to go that far. You know what a great way to kill a particular group of people is? Well, let's take a look at what a group of human military officers decided to do (quoting from a paper of Elizabeth Anscombe's, discussing various logics of action and deliberation):
""" Kenny's system allows many natural moves, but does not allow the inference from "Kill everyone!" to "Kill Jones!". It has been blamed for having an inference from "Kill Jones!" to "Kill everyone!" but this is not so absurd as it may seem. It may be decided to kill everyone in a certain place in order to get the particular people that one one wants. The British, for example, wanted to destroy some German soldiers on a Dutch island in the Second World War, and chose to accomplish this by bombing the dykes and drowning everybody. (The Dutch were their allies.) """
There's a footnote:
""" Alf Ross shews some innocence when he dismisses Kenny’s idea: ‘From plan B (to prevent overpopulation) we may infer plan A (to kill half the population) but the inference is hardly of any practical interest.’ We hope it may not be. """
It's not an ineffective plan.
Machine learning advances are predicated on the internet, will grow the internet and will become what we already ought to know we are. A globe spanning hyper intelligence working to make more intelligence at break neck pace.
Somewhere along this accelerating continuum of intelligence, we need to consciously decide to make thing awesome. So people aim to build competent self driving cars, that way fewer people die of drunk driving or boredom. Let's keep trying. Keep trying to give without thought of getting something in return. Try to make the world you want to live in. Take a stand against things that are harmful to your body (in the large sense and small sense) and your character. Live long and prosper!!!
And in an almost miraculous result, we've managed not to annihilate each other with them so far.
> Income inequality is already rising at a breathtaking pace.
In the US, yes, but inequality is lessening globally.
> The internet has given birth to history's most powerful surveillance system and tools of propaganda.
It has also given birth to a lot of good things, some that are mentioned in a sibling comment.
This has been done many times by human-run militaries; would AI make it worse somehow?
Groups of humans acting collectively can look a lot like an "AI" from the right perspective. Corporations focused on optimizing their profit spend a huge amount of collective intelligence to make this single number go up, often at the expense of the rest of society.
Solders in developed countries no longer want to die en masse.
That's why I found the "outside arguments" here equally important and compelling.
> The outside view doesn't care about content, it sees the form and the context, and it doesn't look good.
If it sounds like and acts like a cult, why should we treat it any differently from a cult? Even if the people in it are all very smart, wealthy, well-dressed, and appear very rational, they're still preaching the end of the world on a certain date. All of those groups have only one thing in common: they're all wrong.
The best rebuttals to all this are the least engaging.
"Dude, are you telling me you want to build Skynet?"
That's not to say that his general premise is wrong, but it's hard for me to take it seriously when his rebuttals are this weak.
Occasionally the crazies are right. Remember when the idea that the NSA was recording everyone's emails was paranoid conspiracy theory talk, except turns out they were actually doing this the whole time.
The fact that the world hasn't ended tells us virtually nothing about how likely the end of the world is, for the simple reason that if the world had ended we wouldn't be here to talk about it. So we can't take it as evidence, at least, not conclusive evidence. Note also that the same argument works just as well against global warming as it does against AI risk.
Turn it around. Suppose there was a genuinely real risk of destroying the world. How would you tell? How would you distinguish between the groups that had spotted the real danger and the run-of-the-mill end-of-the-world cults?
I think that's beside the point. Why would the "aristocracy" owning the factories care about money, when they have all the goods (or can trade for them with other aristos)?
It's not like they need money to pay other people (the destitute masses are useless to them). With their only inherent "capital", --the ability to work-- made worthless by automation, the destitute masses have no recourse-- they get slowly extinguished, until only a tiny fraction of humanity is left.
If you as a manufacturer move to a jobless production system, you gain net margin.
If everybody moves to jobless production, the topline demand shrinks radically.
Yet, for each individual mfg, the optimal choice is jobless production (aka, "loot the commons", aka "defect").
- a military AI in the hands of bad actors that does bad stuff with it intentionally.
- a badly coded runaway AI that destroys earth.
These two failure modes are not mutually exclusive. When nukes were first developed, the physicists thought there is a small but plausible chance, around 1%, that detonating a nuke would ignite the air and blow up the whole world.
Let's imagine we live in a world where they're right. Let's suppose somebody comes around and says, "let's ignore the smelly and badly dressed and megalomanic physicists and their mumbo jumbo, the real problem is if a terrorist gets their hands on one of these and blows up a city."
Well, yes, that would be a problem. But the other thing is also a problem. And it would kill a lot more people.
I make lots of money right now completely via the internet and I am not even breaking laws. It is just probable that an AI at our present level of intelligence could very quickly amass a fortune and leverage it to control everything that matters without humanity even being aware of the changeover.
Ultimately, AI is neither necessary nor sufficient for the powerful to kill the less powerful.
And if it's powerful people trying to build hard military AI, they probably aren't reading LessWrong to understand how to ensure their AI plays nice anyway.
http://voxeu.org/article/parametric-estimations-world-distri...
superintelligence of a military AI is worrisome, but superintelligence of a cantankerous thinking is quite reassuring...
Kind of like what we're already seeing now in Courts, and kind of how NSA and CIA's own algorithms for assigning a target are still far less than 99% accurate.
This is trivially false. Over a hundred billionaires have now pledged to donate the majority of their wealth, and the list includes many tech people like Bill Gates, Larry Ellison, Mark Zuckerberg, Elon Musk, Dustin Moskovitz, Pierre Omidyar, Gordon Moore, Tim Cook, Vinod Khosla, etc, etc.
https://en.wikipedia.org/wiki/The_Giving_Pledge
Google has a specific page for its charity efforts in the Bay Area: https://www.google.org/local-giving/bay-area/
This only includes purely non-profit activity; it doesn't count how eg. cellphones, a for-profit industry, have dramatically improved the lives of the poor.
Also, some of these billionaires are running companies that are great at tax avoidance, probably most of them. Now what? They get to pick and choose where they get to spend there/invest money? I don't buy it.
I believe in wealth, just not this radical wealth separation .
If you raise the marginal tax rate to 99%, then you get rid of billionaires, but you also kill your economy. There are all the failures of communist countries, of course, but even the UK tried this during the 60s and 70s. The government went bankrupt and had to be bailed out by the IMF. Inflation peaked at 27%, unemployment was through the roof, etc.:
What the hell have you done for society?
This story is very similar for most billionaires. They create A LOT of jobs and careers.
Source: https://en.wikipedia.org/wiki/List_of_U.S._states_by_poverty...
http://www.forbes.com/sites/chuckdevore/2016/09/28/why-does-...
A question people don't ask enough is: given that we give vast trillions of dollars to the government, most of which is spent on various kinds of social programs (health care, education, social security, etc), why is there STILL so much poverty, joblessness, homelessness, drug use, crime, and other kinds of suffering in the US?
wow, they're awesome.
It seems possible to me that the technology to turn money into less poverty not only doesn't exist, but that the social structures that make men like Bill Gates rich also make it difficult to create such technology.
Your implicit argument is that todays rich somehow care more about improving society than yesterday's did, which will cause these concentrations of wealth to lead to a different outcome. I'm not sure I see much of a difference between Gates and Carnegie. Different ideas about what the world needs, but not a particularly different approach to capital.
And how are the actions of a few billionaires relevant to what the industry does as a whole? Does Google, Facebook, or, God forbid, Uber, address the problems of poverty and inequality (which are separate problems) as a company?
To a very large extent, charity is irrelevant; charity is a way of buying oneself a conscience without actually changing anything in the world; without even addressing the problems or thinking about them.
The Giving Pledge requires that the money be given to philanthropy, which may improve the lives of others around the world, rather than Californians.
Which is not to say they don't do some good work with it, the Bill and Melinda Gates foundation has done some great work fighting malaria and bringing fresh water to poor communities.
But these same foundations also do a lot of other work, like furthering charter schools which benefits wealthy families to the detriment of poor communities here in the US.
If setting up a foundation that actually helps a ton of people means that their families can get a fraction of that money back, that's fine with me...
Unless you are implying that they are getting more money back through the salaries than they are in donating. In which case i'd love to see a source.
I suppose the poor are no longer subject to long-distance fees during daytime.
There's tons of stuff on this, but, eg., here's a poster from USAID:
https://s-media-cache-ak0.pinimg.com/originals/09/35/2d/0935...
http://www.ictworks.org/2016/06/27/yes-farmers-do-use-mobile...
Etc. And that's just easily measured benefits. There's good reasons why more people in Africa have access to cell phones than to clean water.
"These people are wearing funny robes and beads, they live in a remote compound, and they speak in unison in a really creepy way. Even though their arguments are irrefutable, everything in your experience tells you you're dealing with a cult. Of course, they have a brilliant argument for why you should ignore those instincts, but that's the inside view talking. The outside view doesn't care about content, it sees the form and the context, and it doesn't look good."
The problem with argument ad hominem isn't that it never works. It often leads to the correct conclusion, as in the cult case. But the cases where it doesn't work can be really, really important. 99.9% of 26-year-olds working random jobs inventing theories about time travel are cranks, but if the rule you use is "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity).
That's because it's very effective in practice.
In the real world (which is not a pure game of logical reasoning only played by equals and fully intelligent beings without hidden agendas), the argument ad hominem can be a very powerful way to cut through BS, even if you can't explain why they are BS by pure reason alone.
E.g. say a person A with IQ 110 talks with a person B of IQ 140. The second person makes a very convincing argument for why the first person should do something for them. Logically it is faultless as far as person A can see. But if the person A knows that person B is shady, has fooled people in the past, has this or that private interest in the thing happening, etc, then he might use an "argument ad hominem" to reject B's proposal. And he would be better of for it.
The "argument ad hominem" is even more useful in another very common scenario: when we don't have time to evaluate every argument we hear, but we know some basic facts about the person making the argument. The "argument ad hominem" helps us short out potentially seedy, exploitative, etc. arguments fast.
Sure, it also gives false negatives, but empirically a lot of people have found that it gives more true negatives/positives (that is, if they want to act on something someone says, without delving into it finely, the fastest effective criterion would be to go with whether they trust the person).
This is not only because we don't have the time to fully analyze all arguments/proposals/etc we hear and need to find some shortcuts (even if they are imperfect), but also because we don't have all the details to make our decisions (even if we have a comprehensive argument from the other person, there can be tons of stuff left out that will also be needed to evaluate it).
Cranks lack formal education and spew forth the gobbledygook in reams.
No you don't, you just don't catch it right away, relativity actually holds up under scrutiny. Besides, I reject the premise anyway.
Einstein did serious work on the photoelectric effect first and then gradually worked towards relativity. Outside of the pop history he had very little in common with cranks. This is basically the result you end up seeing when you look into any of these examples used to try and argue against the ability to pattern match cults and cranks, the so-called false negatives never (to my knowledge) actually match the profile. Only the fairy tale built around their success matches it.
So it is with cult-ish behaviour as well, these are specific human failures that self-reinforce and while some of their behaviour occurs in successful people (especially successful and mentally ill or far from neuro-typical people) there is a core of destructive and unique behaviour evident in both that you absolutely should recognize and avoid. Not just the statistical argument that you will gain much more than you lose by avoiding it, it's that it is staggeringly improbably that you will lose anything.
Having established that people evaluate things these two ways, the author then says, "I will demonstrate to both of these ways of thinking that AI fears are bogus."
And yet I suspect that many of the people swayed by one application of the argument won't be swayed by the other and vice versa. Interesting, isn't it?
Meanwhile, rewind ~100 years, and suppose you ignored the luminiferous aether. Or suppose you straight away saw Einstein was a genius? Oh, wait... nobody cares. Because you're dead.
So I'm not sure what the long-term problem is here.
Meanwhile, you, personally, can probably safely ignore people that appear to be cranks.
But in this case, if they're right then we're about to wipe out humanity. That's not safe to ignore.
"What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.
"I'd like to talk for a while about the outside arguments that should make you leery of becoming an AI weenie. These are the arguments about what effect AI obsession has on our industry and culture:..."
...grandiosity, megalomania, avoidance of actual current problems. Aside from whether the superintelligence problem is real, those believing it is seem less than appealing.
"This business about saving all of future humanity [from AI] is a cop-out. We had the same exact arguments used against us under communism, to explain why everything was always broken and people couldn't have a basic level of material comfort."
What nonsense. None of the credible people suggesting that superintelligence has risk are spouting generic arguments that apply to communism or any previous situation.
The question is not IF humanity will be replaced but WHEN and HOW.
Clearly, in a world with superintelligence growing at a technological pace, instead of evolutionary pace, natural humanity will not remain dominant for long.
So it makes enormous sense to worry about:
* Whether that transition will be catastrophic or peaceful.
* Whether it happens suddenly in 50 years or in a managed form over the next century.
* Whether the transition includes everyone, a few people, or none of us.
Hawking, Musk, et. al. are highly successful people with objectively valuable contributions who are known to be about to think deeply and solve problems that others have not.
They are as far from cranks as anyone could possibly be.
Anyone can find non-argument related reasons to suggest anyone else is crazy or a cultist, because no human is completely sane.
What someone cannot do (credibly), is claim that real experts are non-expert crazies, over appearances while completely ignoring their earned credentials.
You immediately self-contradicted here. If a decision heuristic often leads to the correct conclusion, then it's a good heuristic (if you are under time constraints, which we are).
Of course, given unlimited time to think about it, we would never use ad hominem reasoning and consider each and every argument fully. But there are tens of thousands of cults across the world, each insisting that they possess the Ultimate Truth, and that you can have it if you spend years studying their doctrine. Are you carefully evaluating each and every one to give them a fair shake? Of course not. Even if you wanted to, there is not enough time in a human lifespan. You must apply pattern-matching. The argument being made here isn't really an ad hominem, it's more like "The reason AI risk-ers strongly resemble cults is because they functionally are one, with the same problems, and so your pattern-matching algorithm is correct". Note that the remainder of the talk is spend backing up this assertion.
There's a good discussion of this in the linked article about "learned epistemic helplessness" (and cheers to idlewords for the cheeky rhetorical judo of using Scott Alexander and LW-y phrases like "memetic hazard" in an argument against AI risk), but what it boils down to is that our cognitive shortcuts evolved for a reason. Sometimes the reason is because of our ancestral environment and no longer applies, but that is not always true. When you focus solely on their failure cases, you miss sight of how often they get things right... like protecting you from cults with the funny robes.
> "if they look like a crank, ignore everything they say", then you miss special relativity (and later general relativity).
A lot of people did ignore Einstein until the transit of Mercury provided the first empirical evidence for relativity, and they were arguably right to do so.
A heuristic like "it's risky to rent a car to a male under 25" saves a lot of cost in terms of making the decision (background checks, accurately assessing the potential renter's driving skills and attitude towards safety, etc.) and has minimal downside (you only lose a small fraction of potential customers) and so it's a good heuristic.
A heuristic like "a 26-year-old working a clerical job who makes novel statements about the fundamental nature of reality is probably wrong" does reduce the decision cost (you don't have to analyze their statements) but it has a huge downside if you're wrong (you miss out on important insights which allow a wide range of new technologies to be developed). So even though it's a generally accurate heuristic, the cost of false negatives means that it's not a good one.
So, is it a good heuristic to conclude that since crime is related to poverty and minorities tend to be poor, minorities qua minorities ought to be shunned?
Uh ... https://en.wikipedia.org/wiki/History_of_special_relativity#...
This seems, actually, like a perfect argument going in the other direction. Every day, millions of people put cats into boxes, despite the cats not being interested. If you offered to pay a normal, reasonably competent person $1,000 to get a reluctant cat in a box, do you really think they simply would not be able to do it? Heck, humans manage to keep tigers in zoos, where millions of people see them every year, with a tiny serious injury rate, even though tigers are aggressive and predatory by default and can trivially overpower humans.
If you're the first human on an island full of tigers, you're not going to end up as the Tiger King.
One way to get a cat into a carrier - well, the catfood industry created those funny little dry food pellets that are somehow super-addictive. Shake the box, my cat will come. Drop one in the carrier, it surely will enter. Will it eventually adapt to the trick? Maybe, but not likely if I also do this occasionally without closing the carrier door behind the cat.
Yes, we can outsmart the cat. Cats are funny because they do silly, unpredictable things at random, not because they can't be reliably tricked.
I also think this isn't a very good analogy. In this case we're talking about manipulating humans, where we already know manipulation is eminently possible.
Heck it wouldn't even need psychological manipulation. Hack a Bitcoin exchange or find another way of making money on the internet, then it can just pay someone absurd sums to do what it wants.
To become the President, you need only overcome a thousand Florida voters.
To intern the Japanese, you need only overcome two members of SCOTUS (Korematsu v. US)
It isn't necessary for Hawking to be able to trick the average cat into a box. It's sufficient to trick a handful of cats in total.
How well an AI could survive on a desert island is an irrelevant question when Amazon, Google and dozens of others are already running fully (or as near as makes no difference) automated datacentres controlled by a codebase that still has parts written in C. Hawking can easily get the cat in the container: all he has to do is submit a job to taskrabbit.
And Hawking would just ask his assistant to put the cat into the box. You are artificially depriving him of his actual resources to make a weak point.
The first time Random Human A attempts to get Random Cat B into a box, they're going to have a hard time. They'll get there eventually, but they'll be coughing from the dust under the bed, swearing from having to lift the bloody sofa up, and probably have some scratches from after they managed to scare the cat enough for it to try attacking.
However, speaking as a cat owner, if you've iterated on the problem a dozen or so times, Cat B is going in the box swiftly and effortlessly. Last time I put my cat in its box, it took about 3 minutes. Trying for the bed? Sorry, door's closed. Under the sofa? Not going to work. Trying to scratch me? Turns out cat elbows work the same way as human elbows.
The same surely applies to a superintelligent AI?
(Likewise with the Navy Seal On The Island Of The Tigers. Just drop one SEAL in there with no warning? He's screwed. Give a SEAL unit a year to plan, access to world experts on all aspects of Panthera tigris, and really, really good simulators (or other iteration method) to train in? Likely a different story. )
http://www-history.mcs.st-and.ac.uk/Extras/Spitzer_lion.html
This is the one assumption I most skeptical of. In my experience, each time you make a system more clever, you also make it MUCH more complex. Maybe there is not hard limit on intelligence, but maybe each generation of improved intelligence actually takes longer to find the next generation, due to the rapidly ramping difficulty of the problem.
I think people see the exponential-looking growth of technology over human history, and just kinda interpolate or something.
Nick Bolstrom keeps going on in his book about the singularity, and about how once AI can improve itself it will quickly be way beyond us. I think the truth is that the AI doesn't need to be self-improving at all to vastly exceed human capabilities. If we can build an AI as smart as we are, then we can probably build a thousand times as smart too.
You're equating speed with quality. There's no reason to assume that. Do you think an AI will be better at catching a fieldmouse than a falcon? Do you think the falcon is limited by speed of thought? Many forms of intelligence are limited by game theory, not raw speed. The challenge isn't extracting large quantities of information, it's knowing which information is relevant to your ends. And that knowledge is just as limited by the number of opportunities for interaction as the availability of analytic resources.
Think of it this way: most animals could trivially add more neurons. There's plenty of outliers who got a shot, but bigger brainded individuals obviously hit diminishing returns, otherwise the population would've shifted already.
Trust me, if I were to be wired directly to the Internet and had some well-defined goals, I'd be much more effective at it than any of us here - possibly any of us here combined. Because as a human, I have to deal with stupid shit like social considerations, random anxiety attacks, the drive to mate, the drive of curiosity, etc. Focus is a powerful force.
You're equalling human time to AI/computer time. A one day old neural net has already experienced multiple lifetimes worth of images before it is able to beat you at image recognition. It's not trivial, but we just gloss over the extremely complex training phase because it runs on a different clock speed than us.
>The only way out of this mess is to design a moral fixed point, so that even through thousands and thousands of cycles of self-improvement the AI's value system remains stable, and its values are things like 'help people', 'don't kill anybody', 'listen to what people want'.
Bostrom absolutely did not say that the only way to inhibit a cataclysmic future for humans post-SAI was to design a "moral fixed point". In fact, many chapters of the book are dedicated to exploring the possibilities of ingraining desirable values in an AI, and the many pitfalls in each.
Regarding the Eliezer Yudkowsky quote, Bostrom spends several pages, IIRC, on that quote and how difficult it would be to apply to machine language, as well as what the quote even means. This author dismissively throws the quote in without acknowledgement of the tremendous nuance Bostrom applies to this line of thought. Indeed, this author does that throughout his article - regularly portraying Bostrom as a man who claimed absolute knowledge of the future of AI. That couldn't be further from the truth, as Bostrom opens the book with an explicit acknowledgement that much of the book may very well turn out to be incorrect, or based on assumptions that may never materialize.
Regarding "The Argument From My Roommate", the author seems to lack complete and utter awareness of the differences between a machine intelligence and human intelligence. That a superintelligent AI must have the complex motivations of the author's roommate is preposterous. A human is driven by a complex variety of push and pull factors, many stemming from the evolutionary biology of humans and our predecessors. A machine intelligence need not share any of that complexity.
Moreover, Bostrom specifically notes that while most humans may feel there is a huge gulf between the intellectual capabilities of an idiot and a genius, these are, in more absolute terms, minor differences. The fact that his roommate was/is apparently a smart individual likely would not put him anywhere near the capabilities of a superintelligent AI.
To me, this is the smoking gun. I find it completely unbelievable that anyone who read Superintelligence could possibly assert "The Argument From My Roommate" with a straight face, and thus, I highly doubt that the author actually read the book which he attacks so gratuitously.
This isn't just an unreflective assumption. The argument is laid out in much more detail in "The Basic AI Drives" (Omohundro 2008, https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...), which is expanded on in a 2012 paper (http://www.nickbostrom.com/superintelligentwill.pdf).
But it only takes one intelligent agent that wants to self-improve for the scary thing to happen.
If you knew an asteroid would hit the earth 1 year from now, and you had the ability to push a button and become 100,000x smarter, I would hope your values would lead you to push the button because it gives you the best chance of saving the world.
Only if all sorts of other conditions (several of which are mentioned in the post) also apply. Merely "wanting to self-improve" is not enough.
- AI as management. Already, there is at least one hedge fund with an AI on the board, with a vote on investments.[1] At the bottom end, there are systems which act as low-level managers and order people around. That's how Uber works. A fundamental problem with management is that communication is slow and managers are bandwidth-limited. Computers don't have that problem. Even a mediocre AI as a manager might win on speed and coordination. How long until an AI-run company dominates an industry?
- Related to this is "machines should think, people should work." Watch this video of an Amazon fulfillment center.[2] All the thinking is done by computers. The humans are just hands.
[1] http://www.businessinsider.com/vital-named-to-board-2014-5 [2] https://vimeo.com/113374910
Not for long. Robots will be cheaper soon.
> All the thinking is done by computers.
It's hard for humans to operate on more than 7 objects at the same time - a limitation of working memory. So naturally there are simple management and planning tasks that benefit from computers ability to track more objects.
Everyone's probably seen Dr. Strangelove at some point in time. If you haven't, stop reading immediately and go watch it. You will not regret this. Those who have watched it are familiar with a contrived, hilarious, but mostly plausible, scheme by which human beings could be fooled into launching an unauthorized nuclear first strike. This is with technology from half a century ago. As you watch this movie, you will be exposed to a system with checks and safeties that can be bypassed by a determined (and insane) individual. Many humans at every step of the process could have stopped the deployment, but choose to blindly follow orders, well, like machines.
What people should be worried about today is how many humans stand between a decision made by a nuclear power's leader and launch. Humans doubt. Humans blink. Humans flinch. When all the data says nuclear missiles are inbound and it's time to retaliate, humans can still say "No.", and have[1]. If you automate humans out of the system, you wind up reducing the running length of a Dr. Strangelove remake. I suspect it would be down to under five minutes today.
Thanks to popular media, we have this strange idea that taking humans out of the equation in automated weapon systems reduces the possibility for error. Individual humans can, and do, make mistakes. This is true. However, humans fix each others' mistakes in any collaborative process. Machines, on the other hand, only amplify the mistakes of the original user. If a bad leader makes a bad decision with a highly automated nuclear arsenal at his or her disposal, how many other humans will have the chance to scrutinize that decision before machines enact it?
I don't think AI taking over would necessarily be a bad thing. AI could maximize human potential much better than us. I'd rather let our technological offspring take on the role of protector - if it is really smarter than us. And of course it might kill us, but at least we would leave our more advanced children behind, we won't die for nothing.
This is a dilemma every parent has to grapple with - one day we will be weaker and less competent than our children. How will we fare under the care of our children. Will they abandon us, or love and cherish us? Should we avoid children altogether just to be safer? It's just that now humanity has a child, not individual humans.
My best guess is that AI would improve on biological intelligence and make it flourish, not suppress it, because biology is elegant and efficient. The end result would probably be a merge between computer and flesh, silicon and carbon. It won't be us vs them, but a wholly new kind of AI and humans.
Any kind of worrying about theoretical future death is silly, especially accidental death like the kind you are asking us to devote resources to. Why should I care about theoretical future death when there is real, actual death happening all around me?
Heck, if you want to worry about future death, why not worry about climate change deaths, which we can actually model? At least that gives us some chance to do something about it.
Of course I'm assuming you actually care about public health. Maybe you are just worried about your own health, and nuclear strike is relevant to you because it has the potential to pierce through the barriers that distance you from people who are actually facing mortal peril.
Let's try and stop catastrophic climate change, diabetes, and traffic/pollution related deaths, and also allow some people to worry about nuclear holocaust as well.
Maybe not, but there are definitely very physical laws governing everything else, that a superintelligent being's ambitions would run into.
A superintelligent being isn't going to be able to build a superliminal reactionless drive if the laws of the universe say it isn't possible.
More relevant, a superintelligent being isn't going to be able to enslave us all with robots if the laws of chemistry don't permit a quantum leap in battery chemistry.
It is something "AI alarmists" think about.
There are even many things that we already suspect are just engineering challenges. Like rocketry, hacking and robotics. Even if an AI can just act like a million engineers and scientists working for a decade per second, without any higher level of thinking, it could do some incredible things.
Humans are able to calculate plenty with a relatively small amount of mass and energy.
The laws of computational complexity seem like a stronger candidate for a hard limit, but that only means that to get smarter they will require more matter to distribute comoutations across.
The battery limit proposal is just silly.
- gasoline is a liquid battery with much greater energy density than our solid ones
- nuclear batteries are a thing, too
It isn't, but if it wants one then it will likely go and discover the actual physical constraints.
> More relevant, a superintelligent being isn't going to be able to enslave us all with robots if the laws of chemistry don't permit a quantum leap in battery chemistry.
That's what differentiates even a somewhat intelligent AI from technologies we have today. Enslaving us with battery-powered robots is but one way to reach the goal. There are many others. If we stick to robots, you have gasoline-powered robots (gasoline is a liquid battery that's still 30x as dense as our solid batteries). Then you have nuclear batteries, which we know are feasible, but don't exist because of political reasons. And then again, robots are but one way to reach a goal...
But I do agree that, if it has complete information, an AI could potentially beat us at anything that requires thinking.
I believe this is a core misunderstanding. Bostrom never says that a superintelligent AI is evil by default. Bostrom argues the AI will be orthogonal, it's goals will be underspecified in such a way that leads it to destroy humanity. The paperclip optimizer AI doesn't want to kill people, it just doesn't notice them, the same way you don't notice ants you drive over in your daily commute. AIs with goals orthogonal to our own will attack humanity in the same way humanity attacks the rainforests, piecemeal, as-needed, and without remorse or care for what was there before. It won't be evil, it will be uncaring, and blind.
But I don't think this is even possible as when the AI was in child stage and was learning, it would learn from people, so it would become like people. Or at least understand them. At some point it has to know less than a human and will learn at a rate that humans can measure. As we measure we can make decisions about it.
I don't agree with the assumption that a factory making paperclips will make the transition to a super intelligent AI in a short time frame. I think it will take years. And along that route (of years of learning) we'll have time to talk to it and decide if it should be kept switched on.
The thing that irks me about this is how it reinforces a common (and in my opinion, false) dichotomy: either you believe the mind is explicable in terms of ordinary physics or you believe in a soul and are therefore religious. I feel like there should be a third way, one that admits something vital is missing from the physicalist picture but doesn't make up a story about what that thing is. There is a huge question mark at the heart of neuroscience -- the famed Explanatory Gap -- and I think we should be able to recognize that question mark without being labeled a Supernaturalist. Consciousness is weird!
It IS all physical. It's also an unimaginably complex fucking shitload of tiny physical objects working incredibly quickly. If your brain was big enough to see the parts working like little gears of a clock, it would probably be planet sized, or something like that. Huge.
I would EVEN say it doesn't raise any interesting philosophical questions. Are computers silicon, or magic? Is a book paper, or magic? Is the economy magic, or a bunch of people buying shit everywhere?
Knowing that the brain is physical doesn't make me question myself or doubt control over my life or any silly shit like that. Yes all of my actions are technically "predetermined" by the Big Bang, but not in an interesting way, at all.
Semi-related, if you made a giant brainlike calculator out of water pipes and valves and shit, and asked it a question, then turned all its pipes and valves for a while... it would probably just say "please kill me."
If this complex behavior can be explained using only 302 neurons, I have no doubt that the complexity of human behavior and consciousness can be explained using 100,000,000,000 neurons.
It is like searching the ocean for something that by all accounts should be in the ocean, and deciding that perhaps the thing cannot be found, despite having large portions of the ocean still to search. It is too early to make that claim.
I'm not saying that there are not smug philosophers or scientists or just internet nerds that will declare the problem solved, clearly it is all in the brain and there is nothing magical to it but I think they would be incorrect. Modern physics is incredibly esoteric and shares more with meta-physics than what we tend to observe (which classical physics can explain satisfactorily for the most part). I don't think being physicalist is any impediment to having spirituality and and a sense of wonder about the universe and our place in it - the main difference to me is that we can't just accept a deus ex machina and call it a day. That may sound condescending but it isn't a choice for me personally.
https://en.wikipedia.org/wiki/Vitalism
Either everything exists in the material world, or it doesn't. There's really no way of avoiding that binary. If you think there's something else, that is by definition supernatural.
Is anyone exploring this third way at all?
And if you have any piece of evidence, you can be sure the people at the nearest physics dept will love to hear it.
You are not wired up to understand the low level effects, it's simply not a useful. At the same time it does describe how it feels, if we started using firing of C fibers instead of Pain people would assume they understood what was going on.
Bostrom absolutely did not say that the only way to inhibit a cataclysmic future for humans post-SAI was to design a "moral fixed point". In fact, many chapters of the book are dedicated to exploring the possibilities of ingraining desirable values in an AI, and the many pitfalls in each.
Regarding the Eliezer Yudkowsky quote, Bostrom spends several pages, IIRC, on that quote and how difficult it would be to apply to machine language, as well as what the quote even means. This author dismissively throws the quote in without acknowledgement of the tremendous nuance Bostrom applies to this line of thought. Indeed, this author does that throughout his article - regularly portraying Bostrom as a man who claimed absolute knowledge of the future of AI. That couldn't be further from the truth, as Bostrom opens the book with an explicit acknowledgement that much of the book may very well turn out to be incorrect, or based on assumptions that may never materialize.
Regarding "The Argument From My Roommate", the author seems to lack complete and utter awareness of the differences between a machine intelligence and human intelligence. That a superintelligent AI must have the complex motivations of the author's roommate is preposterous. A human is driven by a complex variety of push and pull factors, many stemming from the evolutionary biology of humans and our predecessors. A machine intelligence need not share any of that complexity.
Moreover, Bostrom specifically notes that while most humans may feel there is a huge gulf between the intellectual capabilities of an idiot and a genius, these are, in more absolute terms, minor differences. The fact that his roommate was/is apparently a smart individual likely would not put him anywhere near the capabilities of a superintelligent AI.
To me, this is the smoking gun. I find it completely unbelievable that anyone who read Superintelligence could possibly assert "The Argument From My Roommate" with a straight face, and thus, I highly doubt that the author actually read the book which he attacks so gratuitously.
Near-term AI concerns represent a massive challenge encompassing many ethical and social issues. They must be addressed.
Existential AI concerns, while low probability, have consequences so dire that they warrant further research regardless. These too must be addressed.
There is ample funding and human resources to work on both problems effectively. Why fight about it?
This is a particularly stupid version of https://en.wikipedia.org/wiki/Appeal_to_consequences
"If you don't agree with me, you'll be associated with these people I'm lambasting!" I was surprised to see something so easily refutable used to conclude the argument; the article started out fairly strong.
> If you're persuaded by AI risk, you have to adopt an entire basket of deplorable beliefs that go with it.
Well if they're "deplorable", they must be false! QED.
More likely, artificial intelligence would evolve in much the same way that domestic canines have evolved -- they learn to sense human emotion and to be generally helpful, but the value of a dog goes down drastically if it acts in a remotely antisocial way toward humans, even if doing so was attributable to the whims of some highly intelligent homunculus.
We've in effect selected for certain empathic traits and not general purpose problem solving.
Pets are not so much symbiotic as they are parasitic, exploiting the human need to nurture things, and hijacking nurture units from baby humans to the point where some humans are content enough with a pet that they do not reproduce.
I could see future AIs acting this way. Perhaps you text it and it replies with the right combination of flirtation and empathy to make you avoid going out to socialize with real humans. Perhaps it massages your muscles so well that human touch feels unnecessary or even foreign.
Those are the vectors for rapid AI reproduction... they exploit our emotional systems and only require the ability to anticipate our lower-order cognitive functioning.
If anything, an AI would need to mimic intellectual parity with a human in order to create empathy. It would not feel good to consult an AI about a problem and have it scoff at the crudeness of your approach to a solution.
Even if we tasked an AI with assisting us with life-optimization strategies, how will the AI know what level of ambition is appropriate? Is a promotion good news? Or should it have been a double promotion? Was the conversation with friends a waste of time? Suddenly the AI starts to seem like little more than Eliza, creating and reinforcing circular paths of reasoning that mean little.
But think of the undeniable joy that a dog expresses when it has missed us and we arrive home... the softness of its fur and the genuineness of its pleasure in our company. That is what humans want and so I think the future Siri will likely make me feel pleased when I first pick up my phone in the morning in the same way. She'll be there cheering me on and making me feel needed and full of love.
The idea is if I have a moving agent being able to entertain my dog and keep him occupied, we just need to extrapolate it and imagine robot companions.
Securing love, sex and companionship is hard. with the advent of tinder and likes, women can have 1000's of options at their fingertips while men gotta settle for what they can find. The insane traffic on porn sites suggest that artificially satisfying this need is something we will easily accept. imagine westworld like robots created for human pleasure.
We have this innate needs and desires. Something that ad companies probe and thrive upon.
Google and Facebook are becoming a hoarder of collosal amounts of data, capital and AI talent in the hopes to shove more ads. To satisfy their ever growing share price and do more with less humans. It's a very scary scenario already. Because it's free, all our data belongs to them to endlessly learn about our behaviours and shove ad bait in our face.
We could have a computer program that perfectly simulates the brain, but has some nasty O(2^N) complexity algorithm parts that are carried out in constant time by physical processes such as protein folding. Thus, in theory we could simulate a brain inside of a computer but the program would never get anywhere, even assuming Moore's law would continue indefinitely.
For example compare how many more cities in the traveling salesman problem a super computer can solve vs your grandmas pc. It's more but surprisingly not all that many more.
What do you think of that idea?
Bostrom and Yudkowsky in particular seem happy to hand-wave past computational complexity.
We really have a poor idea of what super intelligence is - none of us can even understand people much more intelligent than ourselves.
Also,
>AI alarmists believe in something called the Orthogonality Thesis. This says that even very complex beings can have simple motivations, like the paper-clip maximizer.
Uh, no. The point of the paper clip maximizer is that it's orthogonal, not that it's simple.
>It's very likely that the scary "paper clip maximizer" would spend all of its time writing poems about paper clips, or getting into flame wars on reddit/r/paperclip, rather than trying to destroy the universe.
You know what can be made into poems about paper clips? Humans. You know what can have better flame wars than humans? Our atoms, rearranged into the ideal paper clip flame war warrior.
>The assumption that any intelligent agent will want to recursively self-improve
That's not really a premise. A better version would be "a likely path to super-intelligence will be a self-improving agent".
>It's like if those Alamogordo scientists had decided to completely focus on whether they were going to blow up the atmosphere, and forgot that they were also making nuclear weapons, and had to figure out how to cope with that.
Yudkowsky has argued that more should be invested in research into AI risk. There are tens of billions of dollars being spent on AI R&D, and somewhere in the tens of millions range spent on AI risk research. Even if advocates wanted us to spend hundreds of millions of dollars on risk research a year , that wouldn't make this criticism fair. You have a point that we shouldn't be ignoring other more important things for this, but to argue against increasing spending from 8 figures to 9 figures you need better arguments.
That Elon Musk, Bill Gates and Stephen Hawking etc are all a little nutters when it comes to UFOs and aliens?
I don't know a lot about UFOs and affiliated cults, but I'm going to guess that those mentioned are not UFO experts just like they aren't machine learning experts.
> That's not really a premise. A better version would be "a likely path to super-intelligence will be a self-improving agent".
Excellent point, but that doesn't give a self-improving agent the ability to ignore computational complexity or the uncertainty of chaotic systems.
If I understand your claim, it's not very relevant. You seem to be saying that, given that superintelligence happens, the probability that it will have happened via a self-improving agent is high.
That doesn't refute the claim that "superintelligence probably will not happen".
> You have a point that we shouldn't be ignoring other more important things for this, but to argue against increasing spending from 8 figures to 9 figures you need better arguments.
Why don't we spend that money on non-vague problems that actually face us now? For example, unbiased AI and explainable AI are topics of research that would help us right now and don't require wacky far-future predictions.
Try this for size: http://milesmathis.com/hawk3.pdf [c.f. "epistemic learned ..."]
Surprisingly relevant XKCD: https://xkcd.com/1170/.
What's more likely? That all those smart people, who in other domains prove their level-headed thinking and good moral compass (past history of Microsoft notwithstanding), have suddenly suffered brain damage, or that the argument against them indeed is another piece in an on-going series of author bashing the techies?
Organized crime and semi-organized criminal gangs stand to establish a highly effective symbiosis with amoral machines which "lack agency."
If a machine wants to kill someone, all it needs to do is find a person to carry out the task in exchange for some benefit such as cash or blackmail ("I will report you to the police for helping me steal the electricity I need, plus the small trafficking operation I helped you optimize last summer").
Two arms and two legs are not what make modern criminals scary. It is their ability to plan, optimize and repeat sophisticated operations. Soon there will be an app for that.
- Up until 2016, Holden held the opinion that it was not a worthwhile concern. He describes why he changed his mind here: http://www.openphilanthropy.org/blog/three-key-issues-ive-ch...
- And he outlines his current arguments in favor of treating it as a significant concern here: http://www.openphilanthropy.org/blog/potential-risks-advance...
In that lengthy write-up, he addresses both the shorter-term risks from things like misuse of non-superintelligent AI, along with longer-term risks from superintelligent AI.
That's always my question for this sort of topic. It seems to me that the only reasonable behavior for someone accepting SAI as a legitimate concern is to advocate defunding all AI researchers, as well as criminal punishments for AI research.
Nice.
All in all, effectively reasoned. I've been making similar arguments for the last few years. AI is likely to create a lot of problems, and solve a lot of problems. But I think both aspects are messy and our relationship with our future technology will be complicated and fraught with regular human issues.
Some of those potential issues are very serious, yes, but serious like automating jobs and not solving the employment issue, or creating a very effective army of automated drones and single-handedly taking over a country (or, sure, the world), not issues of AI destroying the planet and/or enslaving all of mankind.
Is he joking? Yes, it "failed", but in doing so created a wonderful revolution in mathematical thought, allowing exploring a rich area encompassing model theory, types, computability, algorithms, efficiency, and more.
So it's super smart and has its own goals. We can reliably presume that it will need energy to achieve those goals. Will it need to achieve them quickly? Why? Would the superintelligence be shortsighted enough to provoke humans into active combat against it? I see no reason to just assume we know the thing would have human eradication as a goal, need exorbitant amounts of energy and resources because it sees achieving its goals as a terribly time-critical thing to do, etc.
Also, if we're going to build a brain and assume no quantum weirdness - why assume the total absence of a subjective morality? Why assume completely immunity to social influence, which the brains we observe most all encounter?
And let's not forget - human beings are in a weird spot, intelligence-wise. We're smart enough to achieve things, but we're not smart enough to be crippled by the profound lack of control we have over things. We're totally comfortable going out and driving around all day, even though we claim to value human life extremely highly and even though we know that causing deaths of innocent human beings is not THAT far-fetched of an outcome of driving a car. I would be unsurprised if we flipped on a super-AI and after 5 minutes it simply stopped in its tracks, having determined that the probability that some action it would take would result in its own destruction with a non-zero probability and that it is instead taking the safe route and not acting at all. No matter how superintelligent it is, it is not going to be able to magically compensate for the influences of chaos theory which destroy the ability to be certain about ANY prediction. We, as humans, feel very clever with our assumed spherical cows, frictionless surfaces, homogenous air pressure and zero wind resistance. Why would a superintelligence also be comfortable with that?
That depends on whether eliminating us is simple, like stepping on an ant-hill you didn't realize was there, or complicated, like going to war.
I don't buy the argument that for an AI to eliminate us would be relatively simple, that it would be as subjectively simple as we adults find reading and writing, while still involving cognitive effort. That would still require deliberate effort.
The trouble is whether the AI wants, for instance, our electricity infrastructure, and doesn't care about the riots and social breakdown caused when we humans can't get electricity for our own needs anymore. It didn't go into conflict with us. It just killed a lot of people (for instance, everyone on life-support in a hospital reliant on the electrical grid) without noticing or caring.
Likewise, maybe it decides that rising sea-levels and accelerating global warming are good for its own interests somehow. That stuff will kill us, but it doesn't require a complicated strategy, it just requires setting lots of things on fire. Any moron can do that, but most human morons don't want to.
I certainly agree that a "hostile" AI which perceives itself as requiring complicated, deliberate effort to kill all humans will probably not kill all humans. It'll find some lazier, simpler way to get what it wants.
It happened so many times throughout human history, that it's really foolish to reject. People with their own goals burned, raped, murdered. Maybe superintelligence will build its own civilization, and to suppress remorse will put people in reservations.
Because a designer has told it to do this thing "as fast as possible", probably.
> Also, if we're going to build a brain and assume no quantum weirdness - why assume the total absence of a subjective morality? Why assume completely immunity to social influence, which the brains we observe most all encounter?
There is a lot more detail on this point in Bostrom's writings. Social influence is not the natural state of intelligence, it's something that has been built in by evolution. It won't be there in an AI unless we put it in there deliberately.
> having determined that the probability that some action it would take would result in its own destruction with a non-zero probability and that it is instead taking the safe route and not acting at all.
AI is all about making the best possible inferences from limited information - that's literally the whole point of the field. If there is no absolutely safe path it will go for the path it thinks is safest, and pay attention to what's happening. After all, turning itself off is very unlikely to result in its long-term survival.
Exactly how big was this margin? Please someone tell me this was quite a large margin!
More detail in this great summary: http://large.stanford.edu/courses/2015/ph241/chung1/
And the Los Alamos paper is really good reading:
http://www.sciencemadness.org/lanl1_a/lib-www/la-pubs/003290...
I don't know the actual probabilities, but I do really expect a few atoms to get over 1MeV inside a fission bomb (as the reaction emits ~5MeV of energy). But they will almost certainly not hit each other. I also don't see how any atom anywhere can reach 10MeV.
For an artificial mind to have anything resembling a human value system, the argument goes, we have to bake those beliefs into the design.
Because we are so nice to the lesser intelligent creatures of our world? We don't even understand our own consciousness, surely our suffering is not very real and we can be used for meat.
I rather like the smarter people stuff - it's kind of exciting to figure how AI could solve human problems and stuff like the OpenAI initiative seem a sensible precaution against things going wrong. The arguments against seem a little reminiscent of arguments against global warming that say what do all these 'experts' know? I can look out the window and it's snowing as much as ever.
Currently in the name of free services the big tech Co's collect petabytes of data about us and learn from it. The data is owned by them.
So there's maybe a chance AI is extremely dangerous (as in wipes humanity out), and there's a chance that AI might not be dangerous. There are arguments between how likely each choice is. But the more important fact is that since there's so much at stake (all of humanity), we should likely be really really really sure about things.
For example, let's say you live in a house and you've got your kids and your grandkids and all your lovely pets living here. Let's say there's this mystery device that when you press it, there's a small chance that it could be extremely bad. You'd want to make sure it's OK right? Right?!
I agree that we probably don't need to devote ALL of our resources to make sure it's OK, as we can devote some resources to problems that are actually hurting us right now like diseases, etc. But there are a lot of people in the AI community who think that it may be dangerous. It is rational that we'd want to be absolutely sure that it's safe once we AI happens.
We should take the approach of being DEFAULT cautious when talking about any technological breakthrough that have changes that we cannot reverse.
- Dick Cheney
Here is why AI is reason for concern:
1. The world is made up of patterns. Proof: mathematics.
2. Patterns can be distilled down to data.
3. Machine Learning is highly effective at analysing patterns.
4. Machine Learning construction itself is a pattern, albeit a very complicated one, by human scales.
5. Learning speed is bottlenecked by hardware and data availability. Both of which are being solved at exponential rates.
The other knock is that consciousness and its constituent pieces is a product of evolution, which itself is not outside the realm of physics.(I.e. We're sacks of chemicals) This means it is theoretically possible to reproduce intelligence if the same initial conditions are met.
As silicon poses much shorter time gaps than biological substrates, and as equivalent sensory data are fed into simulated environments, the vectors intercepts are quite clear.
I think the key is whether we believe machine learning will reach a critical threshold where it will be able to interpret higher-level meaning.
There is positive evidence that this has to do with layers of neural circuitry, which recent neural net design has confirmed in output.
Additional layers can be added to machines, but it is not so easy to do with humans.
The danger is in autonomy and might. Autonomy being higher level cognition mentioned earlier, and might being the ability to bruteforce its way through problems by sheer speed and iteration, which computer excel at.
It would be great if you guys can spot flaws in this argument. The conclusions are quite grim at my current outlook. It'll be nice to be proven wrong.
Spying is a form of intelligent behavior. ("Intelligence.") So is stealing (and not getting caught, if possible). And, no doubt, hacking [1] is, too.
Wouldn't absolute intelligence mean absolute evil, then?
[1] http://www.reuters.com/article/us-cyber-ukraine-idUSKBN14B0C...
>>My belt holds up my pants and my pants have belt loops that hold up my belt.
>>What's going on down there?
>>Who is the real hero?
I love that he used a Mitch Hedberg joke for this.
I do agree with the point per not doing enough around the immediate impact of AI. The autonomous / electric vehicle transition alone is going rip apart the global economy; preparing for that transition could help midigate it's negative impact on certain populations.
Hawking's Cat, Einstein's Cat: Scams require some intelligence from the victim. The victim needs to mistakenly believe they're doing something smart. Cats are too stupid to scam. Unlike humans they can't talk and they fail the mirror test, suggesting no self awareness. Human behavior when confronted with a deceptive superintelligence is not going to be the same as cat behavior when confronted with a deceptive human.
Emus: The name "Great Emu War" is a joke. The humans had limited resources available, because killing emus wasn't important enough to justify more. If we really wanted to kill all the emus we could do it. We've made plenty of other animals extinct. The motivation for an AI is set by its reward function, which can be as high as necessary.
Slavic Pessimism: This argument suggest that building nuclear weapons is impossible.
Complex Motivations: This isn't obvious nonsense, but consider that all intelligences we've seen so far are the result of evolution, which tends to produce a more complexity than needed. A leg is more complicated than a wheel. A non-evolved intelligence would not necessarily have complex motivations.
Actual AI: Compare with the argument from actual primitive nuclear reactors, which get mildly warm and never explode.
My Roommate: As a human, you roommate had a human reward function. Unsurprisingly, he acted like a human. Why should a non-human reward function result in human-like behavior?
Brain Surgery: Brain surgery would be a lot easier if you could take backups of brains, and duplicate them, and you had no concern about killing the patient.
Childhood: If this turns out to be needed, why can't increasing intelligence result in increased ability to simulate an environment suitable for raising superintelligences?
Gilligan's Island: There's no reason to assume an AI would be isolated. It could have or gain access to the internet and most human knowledge, and the mind architecture could contain many independent sub-minds.
Grandiosity: This depends on assigning ethical value to hypothetical humans, which isn't obviously correct.
Outside Argument, Megalomania, Comic Book Ethics, String Theory For Programmers: Ad-hominem.
Transhuman Voodoo, Data Hunger, AI Cosplay: Why should something be false because it's deplorable? And why should something be false because it encourages deplorable behavior?
Religion 2.0: Ted Chiang talked about the definition of "magic" in an interview (https://medium.com/learning-for-life/stories-of-ted-chiangs-...).
>Another way to think about these two depictions is to ask whether the universe of the story recognizes the existence of persons. I think magic is an indication that the universe recognizes certain people as individuals, as having special properties as an individual, whereas a story in which turning lead into gold is an industrial process is describing a completely impersonal universe.
All religions require some element of magic. Even Buddhism, which is arguably the least magical of all religions, treats consciousness as magic. AI requires no magic, therefore it is not a religion.
Simulation Fever: Simulated universes do not have to be magical by Chiang's definition. A universe could be simulated by something that pays no attention to individuals within the simulation, eg. something that lets a large number of universes run their course then examines them statistically. Possibility of this increases the possibility of living in a non-magical universe despite possibility of living in a simulation.
Incentivizing Crazy: This isn't an argument, it's a description of a field. Perhaps the author meant it to be an ad-hominem: "the idea is false because crazy people believe it".
Only if your scam is too complicated for the victim :). Cats are pretty easy to scam - you just have to stick to simple things. Cats respond predictably to food they like. As a cat person, I find this trick to be more than enough for all my needs ;).
This nitpick aside, I second all your points.
> If this turns out to be needed, why can't increasing intelligence result in increased ability to simulate an environment suitable for raising superintelligences?
Indeed, we could feed the required stimuli to an AI faster than real-time. VR is a thing already, and so is the "fast-forward" button.
He is following the mainstream opinion on this view, so does not have to justify it in detail. Almost nobody subscribes to Penrose's proposal.
> Our best approximations of intelligence are analogous to the comparison between sewing by hand vs the mechanics of a sewing machine, only the sewing machine remains far inferior (at least for now).
There's a blithe assumption of your own, which you don't attempt to justify. Newell and Simon's physical symbol system hypothesis is still taken seriously by mainstream AI, and it has the opposing view: that intelligence is a computational process, and thus we understand many fundamental things about it already.
I'm not taking a position here, just clarifying what the dominant positions are right now.
Also, sewing machines are better at sewing than 99% of people by any metric I can think of, and faster than 100% of people. They also do probably >90% of all the sewing on the planet. So I don't think your analogy does the work you intended.
Read it more carefully, because that's not what he did. He is saying "The AI riskers claim 'If you accept X, then you must accept Y'", where X is "the brain does not use quantum effects" and Y is "AI apocalypse". He does not himself take a position on X, but argues against the X=>Y implication.
And it's not really relevant, but for what it's worth, the majority consensus among neuroscientists is that the brain does not use quantum mechanical effects and can be modeled using classical physics alone. Roger Penrose is in the minority on this question.
I think you may have mistaken the thing I was arguing against for the thing I believe.
1. It’s not as near as you think. Current machine “intelligence” is hardly on par with the abilities of an insect, let alone anything we’d call intelligent. Yeah, it’s capable of analyzing a lot of data, but computers could always do that. We don’t really know how to move forward, and there has been little progress in theory in a long while. We haven’t made any significant theoretical breakthrough in 20-40 years.
2. Intelligence isn’t a general algorithm to solve everything. For example, humans are not good at approximating solutions to NP-complete problems. Other algorithms make a better use of computational resources to solve problems that intelligence is not good at, and super-intelligence is not required to come up with those algorithms, as they use brute force on cheap computing nodes. Intelligence also isn't necessarily good at solving human problems, many of which require persuasion through inspiration or other means.
3. We don’t know what intelligence is, so we don’t know if the intelligence algorithm (or a class of algorithms) can be improved at all. Simply running at higher speeds is no guarantee of more capabilities. For all we know, an intelligent mind operating at higher speed may merely experience time as moving very slow, and grow insane with boredom. Also, we don’t know whether the intelligence algorithm can multitask well to exploit that extra time.
But most interesting at all is what I think is at the core of the issue:
4. Intelligence is not that dangerous — or, rather, it’s not so much more dangerous than non-intelligent things. This is related to point 2 above. We can obviously see that in nature, but also in human society. Power correlates weakly with intelligence beyond some rather average point. Charisma and other character traits seem to have a much bigger impact on power. Hitler wasn’t a genius. But some smart people — because they smarter than average — fantasize about a world where intelligence is everything, and not in a binary way, but in a way that gives higher intelligence non-linearly more power. This is a power fantasy, where the advantage they possess translates to the power they lack.
>It exterminates humanity not because it's evil, but because our blood contains iron that could be better used in paper clips.
The consolation of the cingularity narrative is that it tells a story in which the world is not—yet—being destroyed by paper clip maximizers—in cingularity land, the nightmare scenario is still a ways off, and thankfully, with the help of a few Ayn Rand types, market forces can still save us.
-->source?
> If we aren’t actually living through a simulation, Mr Musk said, then all human life is probably about to come to an end and so we should hope that we are living in one. “Otherwise, if civilisation stops advancing, then that may be due to some calamitous event that stops civilisation,” he said at the Recode conference.
I don't know why we should hope for that because any civilization in a layer in the stack of sims could be destroyed, which would destroy all nested sims beneath.
In fact you should hope you're further up the stack as much as possible because then the probability of destruction is lower.
yet more inflammatory rhetoric from the Pinboard guy
but you will never be a Hell's Angel
Intelligent people know this, which is why there really aren't that many military coups, regardless of how those in the military feel about their political masters.
Have you ever programmed any piece of software that suddenly implemented features of its own? Did your program become sentient?
Laughable. These are child-like fantasies belonging in 50s sci-fi.
What I do know for a fact, is that when it comes to AI fear mongers, there is not much intelligence to be detected.
I'm not sure what particular fear mongers you're talking about, but Bostrom and the like are talking about software that's specifically made to change and improve itself on a fundamental level.
I'd say we're at least 2 major revolutions away from even coming close to what a chimpanzee's intellect, much less a human's.
As a shorthand "quantum effects" are often hand-waved as crazy powerful "weird" things, but we can certainly model them and they really aren't as "magic" as a lot of popular science would have one believe (as much as even some of us otherwise rational people so strongly wish to believe in quantum magic).
Maybe that's the case, but this sounds like it goes against Occam's razor, and give a lot of extra credit to the chaotic process of evolution that it was meaningfully able to harness these quantum effects.
I think you have to be a bit more specific to claim something particularly special.
I think it's much more likely the important difference is that brains are built in 3D allowing far far more connectivity than wiring in ~5 layers of metal between 2D transistors in the same plane does - it's that added complexity we don't have the technology to reproduce (yet) that's the issue
Edit: should state that this is a corollary of "> The Argument From Actual AI," but generalized to any 'intelligent' system, not just neural nets.
http://www.newyorker.com/magazine/2015/11/23/doomsday-invent...
There's been studies done on this type of thing (https://journalistsresource.org/studies/economics/jobs/robot...) and so far use of robots has mostly focused on helping human workers become more productive, and not replacing them entirely. However, for lower skilled workers this isn't always true, and if robots were able to replace even the most skilled workers... that could cause problems for human employment. To quote the article on current status:
"Robots had no effect on the hours worked by high-skilled workers. While the authors found that industrial robots had no significant effect on overall employment, there was some evidence that they crowded out low-skilled and, to a lesser extent, middle-skilled workers."
Now continue that line of thinking and imagine a world where a robot could do any job better than a human...
We could end up with a "is that American made?" or "is that free range chicken?" type of scenario where companies that refuse to replace human workers with robots are competing against other companies that will do anything to lower costs, even if ethically questionable.
So then we potentially end up in a situation where the rich (Executives and stock holders) get richer by replacing costly human workers with cheaper, more efficient robots, and the wealth of the average family declines as people struggle to find work. Alright, well maybe we give all humans an allowance to live off of, food to eat, a home to live in, etc. Except...
Human beings need work. They need to feel a sense of purpose. I don't think the humans from the movie Wall-E hold much appeal. Let's not go there.
Ok, so maybe we pass laws against replacing many human jobs with robots. Well, if the robots are truly intelligent, aren't we then discriminating against a group of sentient beings solely because they are too good at their job? Isn't this just going to be a techno world version of the civil rights and lgbt rights movements?
These are the things I worry about. Not robots killing me.
As a side note, I hope cyborg and other bio tech improves at some point, at lot of these concerns could be mitigated if humans had the potential to improve themselves beyond any normal evolutionary rates.
It's probably the most valid result from the research and yet it is overlooked. We'll have AI capable of replacing the middle class before we have robots capable of replacing the working class.
It's finally available on widescreen blu-ray in Germany but Universal has still not re-released it for the western audience.
The filmed exterior of Colossus HQ is Lawrence Hall of Science in Oakland, CA.
Also, depending on how you measure intelligence, "machines" have been way smarter than humans since the first calculator.
Its hard for me to argue against the fact that this idea isn't eating me.
Our neural nets can drive cars, convert speech to text, recognize images, and maybe even carry on a conversation.
But there's no light behind the eyes. It's all synthetic. It's an emulation which mimes intelligence through the brute force of observing billions of inputs and their associated outputs.
What it decisively cannot do, is something that it never saw before, except by mistake.
Wake me when none of this is true?
We know what intelligence is. And yes, you can actually have dead intelligence, no problem. Intelligence is computing, nothing more.
What we don't know is what is consciousness. You seem to belong to the category of people who believe consciousness is mandatory for intelligence. A lot of us feel differently. Even, there are some who believe consciousness is basically an illusion (I do not share this point of view).
Look at the Chinese Room thought experiment.
Read the novel 'Blindsight' by Peter Watts.
Of course, there's the possibility that a super-intelligence would not be able to break through the human egocentric limitations that may have been built into it by its human creators, and in that case, we're screwed...
Let the thoughts flow as far as you'd like, but come back to reality once in a while. After all, your ideas are conditional to your body.
There are several hackerspaces (https://wiki.hackerspaces.org/Zagreb - I'm partial to the one in Mama), and every now and then we'll gather around a "Nothing will happen" (http://www.nsnd.org/).
There's a vibrant meetup scene (well-covered by meetup.com), and WebCamp Zagreb (https://2016.webcampzg.org/) is the community conference that tries to gather different meetups & communities once a year.
Companies from abroad tend to open development offices here to exploit the cheaper workforce, especially since Croatia has joined the EU. There's also a number of local companies that are constantly hiring, resulting in a solid amount of hiring opportunities even for the part of the crowd that's a bit pickier.
People are leaving Croatia in general, though, and the tech community isn't immune to that. Lots of people moved to other EU countries, and although that's not unexpected at all, I believe it's left a dent within all of the above.
If you decide to visit, give me a shout. :)
Orthogonality is defined by Bostrom as the postulate that a super-intelligence can have nearly any arbitrary goals. Here is a short argument as to why ‘orthogonality’ may be false:
In so far as an AGI has a precisely defined goal, it is likely that the AGI cannot be super-intelligent. The reason is because there’s always a certain irreducible amount of fuzziness or ambiguity in the definition of some types of concepts (‘non-trivial’ concepts associated with values don’t have necessary definitions). Let us call these concepts, fuzzy concepts (or f-concepts).
Now imagine that you are trying to define the goals that will let you specify that you want an AGI to doprecisely, but it turns out that for certain goals there’s an unavoidable trade-off: trying to increase the precision of the definitions reduces the cognitive power of the AGI. It’s because non-trivial goals need the aforementioned ‘f-concepts’, and you can’t define these precisely without over simplifying them.
The only way to deal with f-concepts is by using a ‘concept cloud’ – instead of a single crisp definition, you would need to have a ‘cloud’ or ‘cluster’ of multiple slightly different definitions, and it’s the totality of all these that specifies the goals of the AGI.
So for example, such f-concepts (f) would need a whole set of slightly differing definitions (d):
F= (d1, d2, d3, d4, d5, d6…)
But now the AGI needs a way to integrate all the slightly conflicting definitions into a single coherent set. Let us designate the methods that do this as <integration-methods>.
But finding better <integration methods> is an instrumental goal (needed for whatever other goals the AGI must have). So unavoidably, extra goals must emerge to handle these f-concepts, in addition to whatever original goals the programmer was trying to specify. And if these ‘extra’ goals conflict too badly with the original ones, then the AGI will be cognitively handicapped.
This falsifies orthogonality: f-concepts can only be handled via the emergence of additional goals to perform the internal conflict-resolution procedures that integrate multiple differing definitions of goals in a ‘concept-cloud’.
In so far as an AGI has goals that can be precisely specified, orthogonality is trivially true, but such an AGI probably can’t become super-intelligent. It’s cognitively handicapped.
In so far as an AGI has fuzzy goals, it can become super-intelligent, but orthogonality is likely falsified, because ‘extra’ goals need to emerge to handle ‘conflict resolution’ and integration of multiple differing definitions in the concept cloud.
All of this just confirms that goal-drift of our future descendants is unavoidable. The irony is that this is the very reason why ‘orthogonality’ may be false.
Although the read is ovarall interresting, but I find the style and arguments sub par, too american for my taste.
Seriously, if "it" turned against "us", we'd have the upper hand. For example, quality electronics are hard to make without sending African children down into mines and having African adults shoot each other over the results (ie: conflict minerals). If a superintelligence is reliant on us to propagate its physical-world interactions, we're going to be just fine.
I mean come on, we all work in IT, and we all know just how difficult it is to keep hardware running securely, safely, and in good order. Stuff fails all the time. Similarly, we all know people who are really intelligent, but this doesn't translate to success in life for them.
In short, "being intelligent" isn't enough - the entity also needs ways to effectively work the world around them.
edit: heh, just saw another article on the front page: "IQ is only a minor factor in success"
Nice, but does it fall under political ban?