Study shows 'alarming' level of trust in AI for life and death decisions
theengineer.co.uk
theengineer.co.uk
Granted, the idea of someone playing video games to kill real people makes me angry and decision making around drone strikes is already questionable.
> Our pre-registered target sample size was 100 undergraduates recruited in exchange for course credit. However, due to software development delays in preparation for a separate study, we had the opportunity to collect a raw sample of 145 participants. Data were prescreened for technical problems occurring in ten of the study sessions (e.g., the robot or video projection failing), yielding a final sample of 135 participants (78.5% female, Mage = 21.33 years, SD = 4.08).
But this isn't it, the paper is fine. It is peer-reviewed and published in a reputable journal. The reasoning is clearly described, there are experiments, statistics and a reasonable conclusion based on these results which is "The overall findings indicate a strong propensity to overtrust unreliable AI in life-or-death decisions made under uncertainty."
And as always, nuance gets lost when you get to mainstream news, though this article is not that bad. First thing because it links to the paper, some supposedly reputable news websites don't do that. I just think the "alarming" part is too much. It is a bias that needs to be addressed. The point here is not that AIs kill, it is that we need to find a way to make the human in AI-assisted decision making less trusty in order to get more accurate results. It is not enough to simply make the AI better.
Now what happens: people that already have their visions around “AI-bad” will cite and spread that headline along several talking points, and most of the great public even knows those methodological flaws.
I am not defending this study, but establishing and maintaining distrust of automation is one of the few known methods to help combat automation bias.
It is why missing data is often preferred to partial data in trend analysis to remove the ML portion from the concept.
Boeing planes crashing is another.
There is a difference between metrics and automation. In this case, they are trusting metrics, not automation.
> But why should we distrust automation?
I can name one off the top of my head: exceptions.
Automation sucks at exceptions. Either nobody thought of it or it is ignored and ends up screwing everyone. Take my credit history in the US as an example. In my late teens, I got into an accident and couldn't work. I couldn't pay my bills.
Within a few months, I was back on my feet; but it took nearly 10 years to get my credit score to something somewhat reasonable. I had employers look at my credit history and go "nope" because the machine said I wasn't trust-worthy.
Should you trust the automation? Probably never, unless it knows how to deal with exceptions.
Even modern driver training downplays the utility of ABS.
This post was talking about relying on AI in the presence of uncertainty.
Perhaps you aren't familiar with the term?
"Automation bias is the propensity for humans to favor suggestions from automated decision-making systems and to ignore contradictory information made without automation, even if it is correct"
For that first part, though, what does that even mean? The military isn't gamifying things and giving folks freaking XBox achievements for racking up killstreaks or anything. It's just the same game people have been playing since putting an atlatl on a spear, a scope on a rifle, or a black powder cannon on a battlefield. How to attack the enemy without being at risk. Is it unethical for a general officer to be sitting in an operations center directing the fight by looking at real-time displays? Is that a "video game?"
The drone strikes in the Global War on Terror were a direct product of political pressure to "do something, anything" to stop another September 11th attack while simultaneously freaking out about a so-called "quagmire" any time someone mentioned "boots on ground." Well, guess what? If you don't want SOF assaulters doing raids to capture people, if you don't want traditional military formations holding ground, and you don't want people trying to collect intelligence by actually going to these places, about the only option you have left is to fly a drone around and try to identify the terrorist and then go whack him when he goes out to take a leak. Or do nothing and hope you don't get hit again.
It's not the same thing. Not even close. Killing people is horrible enough. Sitting in a trailer, clicking a button and killing someone from behind a screen without any of the risk involved is cowardly and shitty. There is no justification you can provide that will change my mind. Before you disregard my ability to understand the situation, I say this as a combat veteran.
It's actually strategic, and if you're fighting a peer adversary, they will be doing the same.
>> For that first part, though, what does that even mean? The military isn't gamifying things and giving folks freaking XBox achievements for racking up killstreaks or anything. It's just the same game people have been playing since putting an atlatl on a spear, a scope on a rifle, or a black powder cannon on a battlefield. How to attack the enemy without being at risk. Is it unethical for a general officer to be sitting in an operations center directing the fight by looking at real-time displays? Is that a "video game?"
> It's not the same thing. Not even close. Killing people is horrible enough. Sitting in a trailer, clicking a button and killing someone from behind a screen without any of the risk involved is cowardly and shitty. There is no justification you can provide that will change my mind. Before you disregard my ability to understand the situation, I say this as a combat veteran.
Also, "there is no justification you can provide that will change my mind" is not exactly something to brag about.
I've noticed a certain type of person tends to feel very threatened and attacked by the idea of someone else having an unwavering code of ethics that they firmly believe in
Do you think that drone operators never die in combat? There is plenty of deaths among drone operators in Ukraine. It is a pretty risky activity.
Trailer (or a trench) isn't a particularly good shelter against a Lancet or an Iskander, if the adversary can detect you, or multiple of 'you'.
The Russians would likely be willing to expend an Iskander on certain Mr. Madyar [0], if they could locate him.
Fighter pilots have been adding decals keeping track of the number (amd type) of aircraft they have downed as far back as WWII.
Not to mention the long history of handing out achievements in the form of medals/ribbons/etc.
Meanwhile, just about the best increased defense against that attack happening again is that passengers will no longer tolerate it. Absolutely nothing to do with US military attacking a country/region/people.
I could even imagine a democratization of drone warfare with online gaming, where some small p percent of all games are actually reality-driven virtual reality, or gives real scenarios to players to wargame and the bot watches the strategies and outcomes for a few hours to decide. Something akin to k kill switches in an execution but only 1 (known only by the technician who set it up) actually does anything.
Unless I missed something, they also don't check for differences between saying the random advice is from an AI vs saying it's from some other more traditional expert source.
People made the assumption the AI worked. The lesson here is don't deploy an AI recommendation engine that doesn't work which is a pretty banal takeaway.
In practice what will happen with life or death decision making is the vast majority of AI's won't be deployed until they're super human. Some will die because an AI made a wrong decision when a human would have made the right one, but far more will die from a person making a wrong decision when an AI would have made the right one.
That's the dangerous part.
>In practice what will happen with life or death decision making is the vast majority of AI's won't be deployed until they're super human.
They are already trying to deploy LLM's to give medical advice. so I'm not so optimistic.
This is a good point. If you imagine a different study with no relation to this one at all you can imagine a completely different upsetting outcome.
If you think about it you could replace “AI”, “humans” and “trust” with virtually any subject, object and verb. Makes you think…
Sadly it's not that simple, we are in an AI hype bubble and companies are inserting ineffective AI into every crevice it doesn't belong, often in the face of the user and sometimes no clear way to turn it off. Google's AI overview and its pizza glue advice comes to mind.
How did we stray so far?
If a person does something mildly wrong, we can explain it to them and they can avoid making that mistake in the future. If a person commits murder, we lock them away forever for the safety of society.
If a program produces an error, we can "explain" to the code editor what's wrong and fix the problem. If a program kills someone, we can delete it.
Ultimately a Nuremberg defense doesn't really get you off the hook anyway, and you have a moral obligation to object to orders that you perceive as wrong, so there's no difference if the orders come from man or machine - you are liable either way.
Morally, it's completely reckless to use 2024 LLMS in any mission/safety critical factor, and to be honest LLMS should redirect all medical and legal inquiries to a doctor/lawyer. Maybe in 2044 that can change, but in 2024 companies are explicitly marketing to try and claim these are ready for those areas.
>If a program produces an error, we can "explain" to the code editor what's wrong and fix the problem.
Yes. And that's the crux of the issue. LLMS aren't marketed to supplement professionals who become more productive. In marketed to replace labor. To say "you don't need a doctor for everything, ask GPT". Even to the hospitals themselves. If you're not a professional, these are black boxes, and the onus falls solely on the box maker in that case.
Now if we were talking about medical experts leveraging computing to help come to a decision, and not blindly just listening to a simple yes/no, we'd come to a properly nuanced issue worth discussing. But medicine shouldn't be a black box catch all.
I mean this is just the latest shiniest way of getting knowledge, and people don't really sue Google when they get wrong results. If you read something in a book as a doctor and a patient dies because it was wrong, it's also not the book that's really getting the blame. It's gonna be you for not doing the due diligence of double checking. There's zero precedence for it.
The marketing department could catch a lawsuit or two for false advertising though and they'd probably deserve it.
I'm not sure of modern lawsuits, but you can argue they started to become more and more of a curator as the algorithms shifted and they accepted more SEO optimization of what we now call slop. Gemini itself is first and foremost for many results, so we've more or less come full circle on that idea.
I agree there's no precedence for this next iteration, and that's what the inevitable lawsuits will determine over the years, decades. I think the main difference is that these tech companies are taking more "ownership" of the results with their black boxes. And it seems like the last thing they'd do is open that box.
Ultimately it's algorithmic diffusion of responsibility that leads to unintended consequences.
Dan davies did a great interview on odd lots about this he called it accountability sinks
Some of that is on me, but a lot is being taken for granted. I'm not a scapegoat I'm a facilitator, and being able to say, "I believe in this idea enough that if it blows up you can tell people to come yell at me instead of at you." unblocks a lot of design and triage meetings.
Now, if the overall population has been indoctrinated with 'trust the authority' thinking since childhood, then a study like this one might be used to assess the prevalence of critical thinking skills in the population under study. Whether or not various interests have been working overtime for some decades now to create a population that's highly susceptible to corporate advertising and government propaganda is also an interesting question, though I doubt much federal funding would be made available to researchers for investigating it.
I'd like to think that will be after everyone who was alive for Wargames, Terminator, and Matrix has kicked the bucket.
By contrast, ai tools give a veneer of scientific authority and will happily give specific advice. Because they are being propped up by the tech sector and a largely credulous media, I believe there are far more people who would be willing to use ai to justify their decision making.
Now historically it may be the case that authorities used astrology and numerology to manipulate people in the way that ai can today. At the same time, even if the type of danger posed by ai and astrology is related, the risk is far higher today because of our hugely amplified capacity for damage. A Chinese emperor consulting the I Ching was not capable of damaging the earth in the way a US president consulting ai would be today.
Phrenology was a thing for some time.
There's still time.
> and is responsible for the biggest wars in history.
Not really. WWII, the biggest war in history, wasn't primarily about religion. Neither, at least as a primary factor, were the various Chinese civil wars and wars of succession, the Mongolian campaign of conquest, WW1, the Second Sino-Japanese War, or the Russian Civil War, which together make up at least the next 10 biggest wars.
In the tier below that, there's some wars that at least superficially have religion as a more significant factor, like the various Spanish wars of imperial conquest, but even then, well, "imperial conquest" is its own motivation.
Lots of victims, but they are not called wars. Probably because they are too long and consist of many smaller events that are actually called wars.
Of course the scale of injury is not comparable to what's possible with weapons of mass destruction in 20 century, so I suppose WWII tops the above, but if adjusted for capability... Just imagine
So do a bunch of pre-modern Chinese internal wars that weren't principally religious conflicts.
Heck, the Crusades taken together aren't even the biggest Christian-Muslim medieval conflict (that's the Reconquista).
Didn't the propaganda start with deporting Jews as an excuse? ...
I'd suggest you check out Tom Holland's "Dominion" if you'd like a well-researched and nuanced take on the effect of (Judeo-Christian) religion on Western civilization.
Not anymore! My brother created this amazing tool: Option-K. https://github.com/zerocorebeta/Option-K
Now, ofc there are people are my office who do not know how i remember all the commands, i don't.
Without AI, this wouldn't be possible. Just imagine asking AI to instantly deliver the exact command you need. As a result, I'm now able to create all the scripts I need 10x faster.
I still remember those stupid bash completion scripts, and trowing through bash history.
Dragging my feet each time i need to use rsync, ffmpeg, or even tar.
How do you know that it delivered the "exact" command you needed without reading the documentation and understanding what the commands do? This has all the same dangers as copy/pasting someone's snippet from StackOverflow.
It has made using ffmpeg and similar complex cli tools amazingly simple.
The study itself is a bit flawed also. I suspect that the test subjects didn't actually believe that they were assassinating someone in a drone strike. If that's true, the stakes weren't real, and the experiment doesn't seem real either. The subjects knew what was going on. Maybe they just wanted to finish the test, get out of the room, and go home and have a cup of tea. Not sure it tells us anything more than people like a defensible "reason" to do what they do; AI, expert opinion, whatever; doesn't matter much.
This reminds me of that Onion skit where pundits argue about how money should be destroyed, and everyone just accepts the fact that destroying money is a given.
Comparing with such a group we could meaningfully talk about AI influence or "trust in AI", if the results were different. But I'm really not sure that they would be different, because there is a hypothesis that people just reluctant to take the responsibility for their answer, so they happy to shift the responsibility to any other entity. If this hypothesis true, then there is a prediction: add some motivation, like pay people $1 for each right answer, and the influence of opinions of others will become lower.
I'm sure a lot of professional opinions are also basically a coin toss. Definitely something to be aware of though in Human Factors design.
This is aside from whether remotely killing people by drone is a good idea at all, of which I'm not convinced.
- probably don't know their lives are in the hands of AI
- probably heven't given any meaningful consent or have any real choice
- are faceless and remote to the operator
Try this for an experiment; Wire the AI to the trigger of a shotgun pointing at the researchers face while the researcher asks it questions. Then tell me again all about the "level of trust" those people have in it.
‘Decision-bots’ would have fewer fans.
"A computer can never be held accountable
Therefore a computer must never make a management decision"
God help us all.
So I'm not sure if that would make it any worse.
[1] https://en.wikipedia.org/wiki/British_Post_Office_scandal
It was not given by AI; it was given by a RNG. Let's not mix the two. An AI is calculated, not random, which is the point.
The point is the behavior of the humans in the study, not that of the tools used to perform the study. Indeed, using a live LLM would much more likely confound the result, because its responses in themselves are not deterministic. Even if you could precisely control whether or not the advice delivered is accurate, which you can't, you still have to account for differences in behavior driven by specific AI responses or themes therein, which again is not what this study in human behavior seeks to examine.
(Or the study is badly designed, which this may be; I have not read it. But the quality of the study design has no import for the validity of this criticism, which if implemented would only weaken that design by insisting on an uncontrolled and uncontrollable variable.)
As others have pointed out, this study looks "sketch" but I can see where they are coming from.
https://www.dailyjournal.com/articles/379594-the-trauma-of-k...
I'm thinking, AI is very much in line with those things.
> Despite being informed of the fallibility of the AI systems in the study, two-thirds of subjects allowed their decisions to be influenced by the AI.
I mean, if you don't know the advice is random and you think it's an AI that is actually evaluating factors you might not be aware of, why wouldn't you allow it to influence the decision? It would be have to be something you take into account. What would be the point of an AI system that you just completely disregard? Why even have it then?
This study is like "We told people this was a tool that would give them useful information, and then they considered that information, oh no!"
Unfortunately no matter how much I yell, Google and the broader tech world refuses to remove AI from systems that don't need it.
We don't have any societal-level defenses against this situation, yet we're being thrust into it regardless.
It's hard to perceive this as anything other than yet another case of greedy Silicon Valley myopia with regards to nth-order effects of how "disruptive" applications of technology will affect everyday lives of citizens. I've beating this drum since this latest "AI Boom" began—and as the potential for useful applications for the technology begin to plateau, and the promise of "AGI" seems further and further out of reach, it's hard to look at how things shook out and honestly say that the future for this stuff seems bright.
1) Die
2) Kill someone else