Hacker News Reader

Comment by TheOtherHobbes | Hacker News Reader

I don't get him at all. To me he looks like a Randian dilettante folk philosopher, with a minimal track record of practical achievement.

His AI arguments seem to be of the "If you assume an AGI singularity is possible..." kind, which is suspiciously like assuming god exists, for completely arbitrary values of god, and then debating whether or not he cares enough to hate you.

Most software today sucks so badly it can't tie its own shoelaces without poking itself in the brain and falling over.

How do we know this isn't a general feature of software? Has there ever been a 100% bullet-proof non-trivial software system? Is there anything in the history of CS which implies that such a system is possible?

So suggesting that by (say) 2066 software will not only stop sucking but will be able to meta-improve itself with bug-free perfection to the point where it can eat the world seems curiously ungrounded.

View on HN

You're right about the quality. But in addition AI, as we build it today, is task-oriented, not personality-oriented. People are trying to build cars that drives themselves where directed, not cars that decide whether or not they feel like driving people around at all.

Now, maybe it's possible that a task-oriented AI will at some point develop a personality. But there is no evidence that such a thing is possible.

The "real life" personalities we observe in nature--ourselves--did not evolve from task-oriented machinery. It was the opposite: personality and will is observable in even simple animals; a beetle directs its own actions and tries above all else to preserve its own life. No one sets a task for it.

Where does a beetle get that self-direction? Well, what is life itself? Where did it come from? We find ourselves in very deep waters; the depths of which I think AI-fearing folks don't seem to appreciate.

I don't think it is the case that "AI-fearing folks" like myself don't appreciate what the alleged difficulties of building AGI might be. Rather, the burden of proof is on you, the doubters, to show that we need to spend no money at all on such risks, despite the magnitude of the harm that would be caused if we're right and nothing is done about the risks.

> So suggesting that by (say) 2066 software will not only stop sucking but will be able to meta-improve itself with bug-free perfection to the point where it can eat the world seems curiously ungrounded.

That's a straw man. Software doesn't need to be bug-free to take over the world, literally or figuratively. Cars replaced horses but weren't fault-free. PCs replaced typewriters but weren't bug-free. All software needs to do to take over the world is be sufficiently competent and superior to its competitors - products or (in the case he's concerned about) humans.

Also, it's true that one AI might blow itself up, or make a fatal mistake, while trying to take over the world, but it only needs one idiot misconfiguring his AI in his basement to give us a potential world-destroying calamity - the classic "paperclip-maximiser" problem. That's a very hard problem, and it's bigger than solving the AI control problem - it's no good knowing the secret to keeping AIs under control with probability 0.9999999999999999... if someone else unleashes a dangerous AI without knowing that secret, or deliberately ignores it due to insanity or some death-wish on humanity.

And while he doesn't focus on this, there are plenty of AI risks that we still might conceivably face even if we defeat the outright existential risks, the threats to our very existence as a species. For example, consider what would happen if a paedophile or a rapist got hold of a very powerful AI and was able to command it...

Reply on news.ycombinator.com