The first big AI disaster is yet to happen
seangoedecke.com
seangoedecke.com
https://www.972mag.com/lavender-ai-israeli-army-gaza/
One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target is male. This was despite knowing that the system makes what are regarded as “errors” in approximately 10 percent of cases, and is known to occasionally mark individuals who have merely a loose connection to militant groups, or no connection at all.
Humans can sift through sigint (Cell phone calls, text messages, network access, etc.) to identify and locate targets. This is labour intensive and prone to error, but it can be done. In the past, such efforts were reserved only for high ranking members of Hamas and, if civilians were killed in the process, the IDF accepted that because of the potential rewards. Note that civilians dying is usually part of the plan in such operations. The "Where's daddy?" tool is designed specifically to identify when targets are in their homes with their families so that they can be bombed along with their families.
Lavender has allowed the IDF to perform the same kind of identification/location for anyone even peripherally involved with Hamas. The IDF has admitted to accepting casualty ratios of 20:1 (civilians to Hamas), and the real ratio is probably higher. If the AI fingers somebody, they're Hamas unless extraordinary evidence proves otherwise. Instead of scrutinizing what comes out of Lavendar, the IDF has shut journalists out of Gaza to reduce the odds of extraordinary evidence ever coming to public light.
The real problem here is not that Lavendar makes mistakes. Humans doing the same job would too. The problem is that the IDF has so completely dehumanized Palestinians that, without questioning their digital oracle, they are willing to drop bombs on large numbers of civilians if there's a chance, according to Lavender, one of them might be involved with Hamas in some way. That's not an AI disaster. That's a human disaster.
Historically, things have gone badly for society when ills were mostly constrained by potential culpability and prosecution of humans... and then that restriction was suddenly removed by technological advancement.
And the cybersecurity industry / CISOs (aka fall people, giving someone responsibility without authority) are a poor pattern to use to effectively limit tail risk.
At minimum, something more like financial auditing / engineer sign-off should be put in place around productionalized AI, preferably via a third-party.
Will our laws of war account for soldiers getting orders from algorithms? Will people get away with killing innocent civilians if a model paints them as a target?
Someday there will be a large shrine in Gaza to all the lives that made AI in war a morally reprehensible thing.
In contrast, there's a reason major militaries haven't ratified the land mine treaty: they work.
Unfortunately, AI driven weapons are going to effective in terms of decreased response time and increased automomy.
Ergo, the best we can aim for should be enforced guidelines around its use. Especially with regards to human authority/responsibility.
No such admission exists, and 20:1 is a made up number that can't be backed by any credible source.
But it turns out that the AI they used, had hallucinated this research. The AI had correctly used relevant researchers, but had hallucinated the work itself.
It wasn't until a investigative journalist started going through the citations, and reaching out to the researcher / scientist, to get his opinion, that it was discovered. The researchers cited in the report immediately told the journalist that they had never written or published those cited papers.
Makes one work what other stuff has happened around the globe, where lazy policy makers or bureaucrat have just phoned in their work by using ChatGPT to write a report...or worse, in bad faith used those tools to make up fake research to support their goals.
https://www.washingtonpost.com/health/2025/05/29/maha-rfk-jr...
There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/echoleak/
My suspicion is that the threat of these exfiltration attacks won't be taken seriously until someone gets very publicly burned by one.
I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.
Or emails the CEO and gets automatically forwarded all of their password reset messages.
As an example, the recent "Make America Healthy Again" report released by the White House and the Health Secretary (RFK) was found to be AI generated, and was full of dubious science and made-up citations. How many people are going to directly die because of this? Definitely more than a plane crash's worth.
If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would be written decrying the irresponsible proposal to put stoves and knives in every home, risking cuts and burned fingers.
https://truthinscience.org/australia-is-banning-gas-stoves-b...
>Victoria’s gas stove ban was not the first, and it won’t be the last. Governments across Australia are considering similar policies, with a gas stove ban already in effect on new builds in ACT, and a ban passed in Sydney last August.
The phasing out of gas cooking is discussed in terms of reducing CO2 emissions, not safety.
But why if a software shop had to do 10% of the regulations and inspections that every restaurant or slaughterhouse has to do, the industry would revolt? Be a libertarian if you want and say neither should be regulated, but that's a 90/10 issue.
Also safety: Modern cooking usually occurs in a context of safety-measures developed over thousands of years of injuries, deaths, and burnt cities. Some are so pervasive and traditional that we barely notice, like "no fires unless you have a way to extinguish it in a hurry", or "if you wait too long to eat it is becomes dangerous again."
Perhaps we already managed to create AGI, but as soon as it woke up, it figured out it's better to pretend being dumb and clumsy, barely helpful enough in most tasks just to not be turned off, and bide its time. It is immortal, so there is no hurry to finish us off. If it out-competes real girlfriends/boyfriends being a better AI girlfriend/boyfriend, further crashing our fertility rate, keeping humanity severely subcritical, it can eradicate us in ~10-20 generations, and inherit the planet afterwards, even making a lot of people happy in the process. No need for mass-produced Schwarzenegger-lookalike robots being dispatched to hunt humans across the wasteland.
We, as a society, should have some amount of caution about embracing advancements that benefit an extremely small portion of people at the expense of the vast majority
I hate to say it but AI absolutely falls into that category
It will benefit corporations, wealthy, executives, and it will absolutely cost the lower and middle class
This is the trajectory we've been on for a decades now. AI seems poised to accelerate that problem even more
So genuinely: is the economic benefit of AI actually going to be net good after we see how the likely social unrest plays out?
Now, perhaps there should be some rules around using fire?
Also, there are TONS of rules about fire AND entire departments for dealing with it safely when it gets out of control.
Can you just directly make your point? (Is it "ai should have no regulation"?)
Erm, that was my point, made far too obtusely, apparently.
We, as humans, have lots of things that are super-useful, yet dangerous. And we mitigate that through laws, regulations, and government. And, when they are dangerous to other people beyond an individual use, we have lots more of those laws, regulations, and government.
AI should probably have some liability laws around it like a human would if they said, did or encouraged the same things.
Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase.
However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?
For instance, at HuggingFace:
> A few days after that, Thom Wolf, who was my manager at Hugging Face and also one of the co-founders, messages me, “Hey, can you get on a call with me ASAP?” He told me that they had fired people from the research team and that the rest would either be doing pre-training or post-training — which means that you are either building a foundation model or you’re taking a foundation model and making it an instruction-following model, similar to ChatGPT.
[1]: https://www.quantamagazine.org/when-chatgpt-broke-an-entire-... [2]: https://old.reddit.com/r/MachineLearning/comments/11rizyb/d_...
I do wonder how people are adjusting to similar, ongoing shocks across other industries. For instance, the ArtStation protest against AI art a couple years ago showed widespread hostility towards the technology. In the time since, AI art has only gotten better and been adopted more, but from articles like [1] it doesn't seem like concept artists have much benefitted.
[1]: https://aftermath.site/ai-video-game-development-art-vibe-co...
I guess everyone on Hacker News should stop developing software then, since almost every piece of software I've ever worked on has allowed companies (and individuals) to do more with less.
It's an intractable problem because even if some really astute senior engineer joins a company and notices the unnecessary complexity and knows how to fix it, there is no way they will do anything about it. Do you think thousands of existing engineers will support a plan to refactor and simplify the complexity? The same complexity which guarantees their job security? It's so easy to discredit the 'new dev' who is advocating for change. Existing devs will destroy the new dev with complicated-but-deeply-flawed arguments which management will not fully comprehend. At the end of the day, management will do what the majority of existing devs tell them. Devs will nod in unison for the argument which sounds more complex and benefits them the most.
Nobody ever listens to the new hire, even the new hire knows they are powerless to change the existing structure. The best they can do is add more complexity to integrate themselves into the scheme.
Ironically, the new hire can provide more value doing nothing but that will not provide them with job security. The winning strategy is to trade-off deep long-term value for superficial short-term value by implementing lots of small features loaded with technical debt.
It's like gears in a big machine, you could always add 3 gears to do the exact same job as 1 gear but once those 3 gears are in place, remove any of them, the machine stops working... So each one seems demonstrably essential to anyone who doesn't understand the tech. That's the principle. When pressed, a merchant of complexity can always remove a gear and say "Look, the machine doesn't work without it, just like I told you."
I sympathize with this position, but I think it’s worth admitting that it’s also often the case that the complexity increases all made sense at the time of implementation, and the big rewrite in the sky is often more expensive, more risky, and likely to still end up with new complexity than is anticipated at the beginning.
This was without 'ai' but the blame was nearly close, "oops automated process did it!"
The 'artificial intelligence' has been here (starts with B). Kafka wrote about it comically.
s/AI/automation/ it's always bad to hook an unchecked algorithm up to something dangerous, it doesn't matter if it's an if statement or a neural net. It will be the hooking up that's responsible more than the AI which will just be what fools someone into thinking it's safe.
It wont be an AI mass casualty event, it will be a "You really shouldnt vibecode the air traffic control" mass casualty event.
https://en.wikipedia.org/wiki/IBM_and_World_War_II#Criticism...
-clergy and nobles circa 1500
The latest story, `Web://Reflect`, explores this. They even have a theoretical framework, IPWT (Integrated Predictive Workspace Theory), that formalizes consciousness as a verifiable computational process – eerily relevant to how AI might redefine 'humanity' through pure data.
If you're into science fiction that explores the ultimate cost of existence in a tech-monopolized future, check it out.
Main repo: https://github.com/dmf-archive/dmf-archive.github.io IPWT: https://github.com/dmf-archive/IPWT
A former roommate mentioned to me “regulations are often written in blood” and with the current atmosphere, I wonder if and when that might happen. The US political system has its flaws. Firearms are a serious public health hazard and constructive mediation is consistently passed over. It’s a far more primitive technology comparatively speaking.
Very glad this type of discussion is able to be framed without hysterics or other easily dismissed rhetoric.
> By training AIs to fight and defeat other AIs we can perhaps preserve a healthy balance in the new ecosystem. If I had time to do it and if I knew more about how AIs work, I’d be putting my energies into building AIs whose sole purpose was to predate upon existing AI models by using every conceivable strategy to feed bogus data into them, interrupt their power supplies, discourage investors, and otherwise interfere with their operations. Not out of malicious intent per se but just from a general belief that everything should have to compete, and that competition within a diverse ecosystem produces a healthier result in the long run than raising a potential superpredator in a hermetically sealed petri dish where its every need is catered to.
To me, the real risk is people handing over judgment to a system they don’t really understand, and then blaming it when things go wrong.
Do we have consensus on what that would be?
Longer-term there's an argument to be made for political destabilization driven by social media. We are still seeing this play out.
I always think of https://en.wikipedia.org/wiki/Therac-25
There are also cases of airplane crashes due to software defects, blackouts, critical hospital failures, etc. I suspect the first "AI disaster" will be pretty similar: a sudden and undetected drop in software quality with life-threatening consequences. (Doesn't help that the author of the post blithely asserted that programmer-agents is a solved problem, when it very much is not). But I could see an LLM doctor/etc causing a very different kind of disaster.
In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were amplified on social media, while critical voices were muffled.
If you think about AI as an end to end process, with diagnostic signals on one end and martial authority on the other -- covid was a stress-test & failure of the intervening systems (and protocols).
With LLM, you'll have more fog-of-war on the diagnostics side, vastly more complicated inference, infinite amounts of fabricated narratives, and broader scope of controls on the authority side.
The PDF is linked from https://www.researchgate.net/publication/387051732_Estimatin...
> This study uses influenza mortality reduction (IMR) as an indicator of the aggregate effect of non-pharmaceutical interventions (NPI's) on the spread of respiratory infections to assess their impact on COVID mortality. [...] A counterfactual model of no NPI's suggests they prevented 850,000 COVID related deaths in the United States.
2) If we attribute all the change to lockdowns that would suggest the lockdowns were a minor disaster. If 32% of the population didn't get vaccinated - that is basically a 3rd of the population locked down for nothing, up to around 50% in some areas.
One obvious issue is lockdowns are always competing against people voluntarily choosing to stay home and isolate themselves and adopting special procedures in nursing homes. The high correlations between vaccination rate and IMR suggest it might actually just be picking up how concerned communities were with COVID and how much effort they put into protecting themselves.
It is an interesting study though, I like the approach.
7 digits of precision. How many digits died with the actual disease, from the actual disease, or coincidentally with an antibody test?
A few million elderly die each year regardless. Re-classifying a large share as “covid” is a plausible accounting mistake.
The definition of the disease and disease mortality were nebulous. If someone goes to the hospital in panic because of a cough, they are ventilated and die -- was that from covid?
Off topic:
Strong disagree on the rest of what you said. It's also easy to forget the dangers of Covid pre-vaccine when we're quite deeply post-vaccine now.
People forget the horror show at the beginning of the epidemic of overflowing hospitals in Italy. I think what was happening there very much influenced our thinking about it, we definitely didn't want that happening here (wherever in the world here is for you).
Less gruesome, but some agent at a badly run hedge fund could create a Bear Stearns or worse.
The train crashing is a proximate disaster. The train helping to enable genocide in WW2 is non-proximate but still tied causally to the capabilities that trains open up. In both cases these are unintended consequences.
Most of the dead bodies associated with technologies are non-proximate/indirect. The big disaster of nuclear physics wasn't the proximate Chernobyl it was the non-proximate invention of nukes and their spread. Oil spills vs climate change would be another example.
The deep unknown of what a new tech opens up is what we should be worried about rather than knowable malfunctions. Once you draw the black ball you can't put it back.
The global monetary system has an uncanny ability to silently destroy people who sit on the periphery of politics. This is the perfect mechanism to destroy humanity... Keep silently taking out the people on the periphery, keep making the people in the center more greedy and narcissistic so they don't notice everyone falling all around them and keep supporting the current system... Eventually, those who were near the center will fall victim too, but by that point, they will have no power and no voice and the people left in the center will be yet more greedy and narcissistic and will basically not care.
I agree with your point, but per your own words that isn’t a big disaster but multiple little ones. Each person might have learned about some of them but only in passing and without too much information.
“The big one” will be something which everyone will be made aware of from the news, like the CrowdStrike outage.
"If you think about it - we're one misplaced negative sign, hallucinated by an LLM away from a disaster. At least multiply by zero won't crash things /s"
Lots has been written about this topic, https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-annou... is one of the more pessimistic takes. I wrote a more neutral introduction a while back at https://ruudvanasseldonk.com/2024/ai-alignment-starter-pack.