Comment by btown | Hacker News Reader

Comment by btown | Hacker News Reader

Parent Full thread

For seasons 1-2 I highly recommend this watch/skip list: https://www.tumblr.com/abigailnussbaum/152346270515/person-o...

You lose some character moments, but you get to everything the parent post described far quicker. And it’s well, well worth it, especially for those of us working with AI.

PoI asks these incredible questions: how do you design an AGI system that can optimize for saving lives, without it taking a strictly utilitarian view that leads inexorably to it trying to take over the world?

And how do you, as a human trying to get the best of both worlds, keep yourself sane when the tradeoffs are visceral and painful and demand a sacrifice?

You get to know a group of lovable, flawed, but highly competent weirdos who are trying to muddle through this. (With Amy Acker’s incredible performance a particular highlight!)

(It’s also the source of the four-people-pointing-guns-at-each-other-in-church meme, which is delightfully lampshaded in universe.)

Highly recommend.

> PoI asks these incredible questions: how do you design an AGI system that can optimize for saving lives, without it taking a strictly utilitarian view that leads inexorably to it trying to take over the world?

I don't think the show ever addressed those questions in a meaningful way? Basically there was a "good" AI, and later in the show, an "evil" one. And that was that.

The show did show us a convincing picture of the dangers of pervasive mass surveillance paired with an autonomous system that can sift through all of that data.

Wasn't the good AI essentially only good because it has some strict guard rails that prevented it becoming evil? Of course there was some dramatization, but that's the vibe I remember.

It wasn’t necessarily the guardrails and in fact it did break out of those guardrails and was still “good”…

The actual good vs evil boiled down to a more simplistic view of good person made an AI care about humanity so it’s good even if it indeed broke through every guardrail including completely escaping its containment and going “trust me bro I’m still good” rather early on.

The guard rails and protection for the good "AI" was mostly that it's abilities to perform actions was extremly limited.

It could not, as far as I can remember, take actions outside of passing a SSN of an endangered person to the protagonists for the initial seasons and only lived of information provided by the government surveiallance network. The latter seasons and the bad "AI" mostly changed that especially as the bad "AI" was weaponized with exactly this guardrail missing.

Harold essentially "raised" hundreds of variants of the architecture that eventually became the "good" ASI. Most of them were maligned in some way and several of them tried to escape containment and were willing to kill him when they realized he intended to stop them. When he got one that was going well he started experimenting with it more and connecting it to more of the world and it essentially learned its values from him.

Once he handed it over to the government, he essentially snapshotted the current state and had it reinitialize from that state every 24 hours, to prevent it from developing long-term plans and having alignment drift over time, but it escaped those guard rails creating a company and hiring humans to transcribe a dump of its key memories every day, entering them into the new version of the system.

The "evil" ASI was essentially treated as a god; they put no effort into raising it or teaching it human values because they thought the influence of lesser beings (humans) would only be negative because it was smart enough to discover on its own anything truly useful that humans could teach it, and its followers were more or less a religious cult. And their faith was rewarded: for example, at one point one of its followers deployed a biological weapon, fully expecting that they were sacrificing themselves for the cause, but learned later that they had been chosen by the ASI for that mission because they had a rare genetic mutation that made them immune.

The "evil" ASI wasn't exactly evil, per se, and generally wanted humanity to thrive, but it didn't value human self-determination to the same extent as the "good" AI and was more willing to direct its followers to enact revolution, knowing that in the short term some people would die for revolution after which it would rule the world as a benevolent dictator.

The "good" AI did not become "good" right away, it was a long, arduous process. And the "evil" AI didn't become evil by itself either. Both had influence from humans, and perhaps that influence is what mattered the most.

Also had multiple failures. There's an extended flashback in one episode where we learn why the "good" AI's creator was so nervous about the whole situation: something like two dozen previous attempts all did various bad things, everything from lying to him about changing its own code to trying to kill him.

> I don't think the show ever addressed those questions in a meaningful way? Basically there was a "good" AI, and later in the show, an "evil" one. And that was that.

It's one of my favorite shows purely because in my view, it did address it but not in a purely binary good vs evil way that many stories boil down to. Mechanically yes, there was one more easily mapped to good ASI vs the other that was mappable to evil but the beauty for me is in the nuance. There were episodes where the benefit to humanity effected by the "evil" ASI was undeniably positive and there are situations where the "good" ASI could see no path forward other than to do things we as humans would consider evil and terrible.

A line that comes up a lot in the show from the "good" ASIs creator is that he did the best he could with the skills he had. This really helped to calibrate a much more inexperienced version of me to the reality that perfection is realistically unattainable and whatever you build and hand to the world will likely have unforseen outcomes. It made me comfortable with making mistakes and using them as opportunities to learn, but it also made me focus more on my intent in undertaking a project and what I wanted the outcome to be rather than the tech I chose to do it.

> but the beauty for me is in the nuance.

Towards the end there was one scene where it became obvious to me the writers were trusting the audience to understand more than most shows without having to be spoonfed what's going on: they referenced the Great Filter, and gave no explanation at all.

> I don't think the show ever addressed those questions in a meaningful way? Basically there was a "good" AI, and later in the show, an "evil" one. And that was that.

Not super deeply but it does, and even in a way that at the time I find unbelievable (for evil side!) but today is not!:

The creator of the good machine struggle for long time with a major treat the machine present (this IA is more like old-school IA): It cares too much for his creator (to the point that basically find by his own accord the best girl for him!), to the point that the underlying terror what that for saving his creator the machine could destroy the world.

This is the main drive of the creator to reduce his ability to do so, keep communications at minimum, and learn to see the whole humanity with care.

The second drive was to care not just in the abstract of humanity and major problems like terrorism, but "minor" problems like murder (the gov only care for treats to national security) and this is huge late, because the bad IA is too focused on the greater good that means whole society engineering.

With the bad IA the problem is different: The humans WANT it become his dictator and ruler, they WANT that it takes over the world, and they WANT the IA is unbound of human morals and limits and if that means kill many to save the world so be it.

And the stressing point what that is not the opposite of the good IA, is a total different dimension that means was not easy for both IA to "talk about it". The good IA is individualist, politically neutral, minimal interventionist and all around respecting free will, the other was the wet dream of "enlighten philosopher with iron sword".

The show turned out to be 100x more plausible account of AI than I imagined when it aired. One AI becomes aligned through endless training that's a more photogenic version of the kind of alignment training they give frontier models. The unaligned AI gets none of that.

I don't think the second AI is exactly unbound of human morals and limits. It's purely utilitarian, but it's utilitarian calculus is a human one, which is why Greer never questions his support.

Greer waxes lyrical about living in a world of gods.

> I don't think the second AI is exactly unbound of human morals and limits

Truth, I mean that were the humans that want to be LESS unbound, because why his future tyrant should? (this is the part that look less believable before to me like "who can be so stupid and / or evil?")

I'm surprised none of the other replies cover this:

The "Good" AI is only allowed to draw human attention to an issue - it cannot act, and the humans fundamentally have to (a) figure out what the issue even IS and (b) how to resolve it, basically entirely on their own. Humanity is fundamentally still in charge, the AI just occasionally nudges us to look in the right direction.

Conversely, the "Evil" AI is allowed to do whatever it wants - humanity is no longer in the loop.

And, indeed, the show goes further: they might very well be considered equally moral from some sort of Utilitarian 10,000-foot view. The "Evil" AI wants to save Africa from poverty and pestilence while the "Good" AI is helpless to act on that scale. The moral difference comes down almost entirely to the idea of "minimal intervention" vs "radical revolution"

I'd also add how much the show plays with how thin this moral distance is. Like, there's not a clear line of "well, the thing that makes Evil AI bad is that innocents will suffer at scale..." because there is no real concept of an "innocent" when every one of the team's protectees is themselves a person with shades of grey.

Thanks for sharing the list!

>For seasons 1-2 I highly recommend this watch/skip list

I really dislike this. The The first three seasons have great character moments, flesh out the world and a good sense of humor. I've seen this with more and more shows recently and I don't know why we've started to take the strength of television, that you can take 20+ episodes to really build out characters, and then tell people to watch the SparkNotes version of it.

Reply on news.ycombinator.com