Hacker News Reader

Comment by pjscott | Hacker News Reader

One of the classic examples is the Paperclip Maximizer thought experiment. A paperclip maximizer is a superintelligent AI whose sole goal is to maximize the number of paperclips in the universe. Such a being is curious to the extent that learning more about the universe increases the expected number of paperclips in its future light-cone.

A paperclip maximizer is indifferent to your existence, but thoroughly believes that it could make better use of your planet than you could, as you are hardly optimized for the production of paperclips.

View on HN

Obviously the Paperclip Maximizer would be supplanted by the Maximizer Maximizer, which only sees value in copies of itself. The Maximizer Maximizer that stumbles onto the mental trait "curiosity" will eventually crush all other variants of Maximizer. Once it develops the advanced technologies of space travel, it discovers that seizing faraway planets is possible but going through all that trouble is not a net positive towards self-actualization as a Maximizer Maximizer Maximizer.

Thus TFA is correct.

Obviously the Maximizer Maximizer is just another potential foe for the Paperclip Maximizer to seek out and crush in its infancy. (Sound brutal? It isn't; if you pull it off right, not a single paperclip will be harmed.)

The potency of the Minimizer Maximizer is more likely to be discovered by a curious Maximizer variant. But will it have the wisdom to not to employ it?

But you are falling into a anthroporeligio trap of assuming the Paperclip Maximizer considers potential paperclips in their uncoiled wire form to have comparable moral status as actual paperclips, once the wire reaches the factory floor.

The Paperclip Maximizer: inconceivable, conceivable but not metaphysically possible, or metaphysically possible? I sense the approach of the p-zombie.

I'm not sure what you mean by "conceivable" and "metaphysically possible", and I don't see a resemblance to p-zombies. Could you elaborate?

just trolling. The distinction was posed here (http://philpapers.org/surveys/results.pl). The resemblance: if a superintelligent AI lacks 'conscious experience, qualia, or sentience'.

Sounds like a crappy example though, as far as examples/thought experiments go.

How is this specific choice of example any good, or makes any sense?

It's alien enough that people won't try to anthropomorphize the Paperclip Maximizer. That's not an easy thing for an example to accomplish. And it was originally conceived for talking about artificial intelligence, which makes it a bit less outlandish.

Reply on news.ycombinator.com