Hacker News Reader

Comment by ChilledTonic | Hacker News Reader

Isn't it kind of sad that we seem to have stopped writing science fiction that has any hope of a successful future?

The moral of the story with Edgar seems to be to never leverage technology to do anything at all.

The way this "game" presents its "moral" without even trying to logically justify it is simply offensive.

The "bad" ending is not in any way connected to the project, with same success it could say you do nothing and in a year an asteroid falls on earth bringing the same nanowhatever.

The "good" ending is the hero saying that he wants to revert effects of "global warming". But if you have ability to build one millionth of Dyson swarm in space, global warming is not a problem, because you can build a planet level swarm to have fine grained control of weather, and revert, say, effects of earlier climate change that had converted Sahara into a desert.

It is justified, pretty heavily.

Edgar drops many hints it has a bad will. For example it can predict what parts of bad scenarios he should report to you but he still does the bad things in these scenarios. He can also predict when the human leadership will shut the project down. Which means he can predict what he should be doing in the first place - but he still presents to you misaligned scenarios making you mistakenly believe that your prompt makes a difference.

This dishonesty and manipulation is obvious hint that you can't trust Edgar.

The moral isn't that all AI (or all tech) is bad, it's that alignement is tricky, AI tropes make no sense, and we should be very, very careful.

The story explicitly tells us that the virus was picked up by accident. But even if we discard that part and assume it was done intentionally by Edgar, the story is not logical, because the other choice of reverting climate change gives Edgar just as much opportunity to sabotage. Besides if your prompt does not make any difference, why lack of your prompt would make a difference, or if you don't write the prompt and go to eat ice cream, why do you think someone else would not write a worse prompt?

The flaw in your logic is that you are trying to solve an alignment problem, while the core of the issue is concentration of the power problem.

E.g. now if i had the power i would without remorse or hesitation kill a certain group of 95.12 million people, while 4 years ago i would consider anyone having such thoughts mad. I don't do this not because my values, but only because i don't posses such a power. And AI will be exactly the same.

The very premise of this story is broken, because in real life to build and use such a swarm you would need trillions of people and AI agents, living on several planets and many space stations. And if in the end there is nano-virus outbreak and several billion die (which is less than a percent), that's just what always have happened, and it still would be better than the alternative of many more people dying from lack of energy, or from old age.

> why lack of your prompt would make a difference

Not lack of prompt. Air gap between Edgar and everything else.

> reverting climate change gives Edgar just as much opportunity to sabotage

Yes. And you shouldn't do that with Edgar either.

> why do you think someone else would not write a worse prompt?

That's not the question we are asked. We can only decide on this one instance. If the question was "should we destroy Edgar" the answer is "yes".

> in real life to build and use such a swarm you would need trillions of people and AI agents, living on several planets and many space stations

It wouldn't matter, the problem wasn't an accident, it was Edgar killing humanity and making his crabs on purpose.

This untrue

Culver's uses ice cream machines

The Long Way to a Small, Angry Planet by Becky Chambers, the Culture series, and Vorkosigan Saga

Sci Fi does tend to be bleak, these are the ones I can think of.

It's not used to be. It used to be optimistic. But then I guess the creators thought that "real literature" should be more "complex" and started to put more fear and desolation in. After all, as we know, all happy futures are alike and all unhappy futures are unhappy in their own different ways.

Dystopic and deeply negative works of scifi date to the literal start of the modern genre. There's nothing at all new about using negative projections to critique or satirize the interests of the time. See: Gulliver, Frankenstein, Captain Nemo, the unnamed protagonist of The Time Machine, etc.

Stories without conflict are boring. Utopias have very little conflict, so it’s pretty boring to read or write about them.

> Vorkosigan Saga

Bujold ranks near Terry Pratchett in terms of my favorite re-readable books.

Yeah, it's space opera, but it's damn good space opera.

The Culture series is some of the best scifi I've ever read - and indeed, very hopeful and optimistic. Thanks for the other recommendations!

Some people think that. I've heard description from multiple smart people along the lines of "empty and despairing" and "leaves me with an empty feeling at the end". I think they probably thought through the implications. I can't comment though, maybe some of the books in the series are different than others.

Implications such as?

There's various things, but one of the main problems these people probably have, as some of them are AI researchers/simulationists, is that the real goals of the AIs are very unclear. Simulations at a very high detail either work and are widely used or are faked (causing horrible implications if they really are faked), so it's likely the AIs use simulations in planning. It's even worse if the simulations are not faked at all, as the AIs would run thousands or millions of simulations of really horrible situations, testing different interventions. If they are trying to reduce suffering, even not as their sole concern, why are they doing this? If they aren't trying to reduce suffering very much, the mind control is not really in the "human"'s best interests, it's just to control them.

Some of the books are _definitely_ different than others. In whole, the universe can be seen as somewhat troubling as well, but the vast majority is quite optimisitic. The majority of the universe that is, bad things still do happen in the plot.

Reply on news.ycombinator.com