Hacker News Reader

Comment by callum85 | Hacker News Reader

I don't see how a ban could work. We already have technology that can identify targets, so the issue is whether a human is involved to approve each kill decision. But what constitutes human approval? Is it enough for a human to sit and watch target details flashing up on a screen, and intervene if they see an incorrect target?

What exactly would be banned?

View on HN

We have bans on chemical and biological weapons. You don't say "but everything is made of chemicals. What are we even banning?" The law will create a legal definition and courts will interpret it. It's not the hard.

This is something of a problem. The system which scans mobile phone metadata (among other things) and turns it into drone bombing targets is already enough like the latter. A human is involved in pulling the trigger, but really they're following orders rather than having made the target/not target decision. And they may not even have the security clearance for the information on which the decision is based.

I don't think that's so much a problem. The problem is literally robots pulling the trigger. What happens when the president is targeted due to error/hacking? A member of the military would have a major problem with assassination of her own leader; a robot may not care. What happens if everybody becomes a target? The military personnel would figure things out real quick, a robot, again, may not care.

A member of the military might not have any problem at all with the "accidental" assassination of their leader, whether that leader is the president or the company sergeant. AI could easily provide plausible deniability.

I've been very critical of claims of the dangers of AI, but this isn't about AI - this is about giving lethal weapons to autonomous systems.

And that's just stupid. All software has bugs, and even if you don't have a problem with autonomous killing systems - I certainly do - it's incredibly irresponsible to build systems that could kill just because someone left out a semicolon or couldn't convert between miles and km.

It's taking the "blue screen of death" experience just a little too literally.

Umm, we already have machines that can kill people if someone "left out a semicolon". Airplane autopilots, medical equipment, car computers, etc. Not saying this makes it OK, but that isn't the reason. Life-critical code isn't a new thing.

And those systems all have humans operating in conjunction with them, none of those things are (currently) autonomous. The success thus far of life-critical code usually depends partially on the assistance of humans.

Most of those systems operate on their own unless/until they hit something they can't handle and a human is standing by as failover. So, unless you're saying it's OK to have deathbots as long as a human is on hand as failover if the deathbot gets confused, I think my point stands.

IMO, the current state of the F-35 rather suggests it doesn't.

A member of the military would have a major problem with assassination of her own leader

https://en.wikipedia.org/wiki/List_of_coups_d%27%C3%A9tat_an...

Usually if a leader is assassinated, the assassin is a member of the country's military.

The US is comparatively rare in having had no coups since independence and only one civil war. The US also has very strong indoctrination against this kind of thing, but individuals are just as vulnerable as robots to making the wrong decision if they are fed false information. Such as endless attacks on the legitimacy of the president's election, birth certificate, allegations of being a foreign agent, etc.

The US is of course not above assassinating foreign leaders, conspiring to have them deposed, or providing arms to military coups against democratically elected governments. Allende is probably the biggest example.

Edit: and of course there's always the risk of a single rogue individual, from Dr. Strangelove's " Brigadier General Jack D. Ripper" to the very real Ford Hood massacre. In fact, Dr. Strangelove should be required viewing on this subject; not only is it darkly hilarious, it's all about setting in motion destructive partly-automated systems which cannot be countermanded.

It's funny, because probably is more relevant to think the problem the other way.

What happens when leaders know that, no matter what, never are going to be "targeted" by their own automatic military?

You've had drone operators coming forward and exposing problems with the "program". That's a very good reason to keep humans close to these tasks.

Reminds me of The Machine in Person of Interest.

(Spoiler alert for those who haven't seen the first 3 seasons.)

The Samaritan works like you described - turns data and metadata into kill decisions. The Machine just outputs a social security number of a "person of interest". Figuring out why is the job of humans.

The show features an interesting discussion about what happens when an AI following the first method (data -> kill decision) starts lying to you to further its own agenda. The same issue applies when you have a black box of sub-AI complicated analysis algorithms that make the decision.

Why in the world would you design such a system that deliberately fails to explain its decisions?

Classification.

If the system is using all available classified information, then an explanation of its output would have to include the security classification of every input, and the system of classification may also be such that no one person is cleared to know everything. Therefore nobody can know the rationale.

(I've made this up, but I think it's the most plausible way such a thing would get built, and it's the logical outcome of "no read up/no write down" security rules.)

I don't know much about the exact structure of classification. The situation you described would be certainly possible if the classification system is not a tree but a general graph. Is that the case with clearance levels?

Yes, it's designed that way for counter espionage reasons. In theory only the president gets full clearance for everything.

In the show, Samaritan is a full-fledged AI with its own goals and agenda.

In terms of the real world, how many systems do explain step-by-step how they arrived to their conclusions? With some of the analysis and reasoning algorithms, it would be unfeasible to require humans to double-check every step. How would you quickly tell if a Bayesian network aggregating probabilities from hundreds of observations is correct?

We don't even require explaining themselves from humans. Instead, we have the concept of trust, which is something one earns. As a commander, I will trust that my subordinate is doing his analysis in good faith, but because there is abuse, we try to have an audit trail that can be used at later date to detect who acted wrongly, and should be deemed untrustworthy. Especially in combat situations, there is no time for an individual to evaluate every piece of evidence.

So the point is - are you willing to trust the algorithms you don't understand? With humans you at least share the common mind architecture. In the show, Machine was designed to avoid this question altogether by refusing to make decisions that carry moral weight, only to point out potential persons of interest for human agents to investigate.

It's a plot device. The characters spend their time trying to understand what is going on.

But in the context of the script: the designer wanted it to be a 'black box' so nobody controlling the machine used it for it's owns goals. The machine only outputs the minimum relevant information. It's not a bad show.

There is well known problem with neural networks - they cannot explain reasoning behind their decision. In fact, there isn't any reasoing there.

The advantage of humans sitting and watching target details flash up on screen is that not being in any immediate danger themselves they should have fewer excuses for misindentification. Technology should make humans more acccountable for kill decisions rather than less.

In practice it would just make it easier to kill -- so more killings.

In wars, invasions, etc, including those done by "democratic powers" few care about "misindentification" (or even far worse offenses to the enemies including civilians). That's why the mostly go unpunished or just get a slap on the wrist.

"Kill the all and let god sort them out" is the modus operandi.

Reply on news.ycombinator.com