Actually delivers on the attention-grabbing title. Comments also worth perusal.
It simply suggested that the person follow a series of steps, the culmination of which could have been serious harm or potentially death.
If that difference is meaningful to you, then good for you
However in most countries the justice system works differently. If your neighbor gives you this advice and it kills you, what will matter the most is not the authoritativeness of the advice (he's not a food expert, he was just talking) but the intent. If it also turns out that he had an affair with your wife and mentioned he wants to get rid of you, then he'll very likely go to jail. The "he fell for it and took my advice as if it was authoritative information to be relied upon" defense won't work.
Here (hopefully) the intent is missing. However the whole reasoning that advising partly bears no responsibility for texts it "generates" doesn't work very well.
I'm not saying that users are the only responsible parties here. I'm saying that the humans, including both the users and Google ('s managers and employees), are the only parties that can possibly be responsible. The LLM is not responsible. The LLM is incapable of having intent, as far as we currently know.
I've never used Gemini so I don't know how the user interface comes across, but it might be completely appropriate for Google to be facing liability lawsuits over things like this. What doesn't make sense is narratives that push any of this shared blame onto LLMs or "AI" itself, when the real world effects hinge entirely upon how humans choose to attach these algorithms to the real world.
Though I also don't really have problems with headlines like "AI tried to kill a man". True LLM can't be held responsible, but so are e.g. animals. If an unleashed dog kills a man, the owner will be responsible but the headlines will be still about the dog.
Then it can be argued that LLMs don't have intent while the dog had, and many more analogies, arguments and nuances will follow. My point is, this headline is still at an acceptable level of accuracy, unlike many other headlines that distort the reality completely.
Society has no such general understanding for LLMs. If we did, we wouldn't have situations like this in the first place, because people wouldn't be expecting them to generate instructions that can be followed verbatim. Instead we get the marketing buzz of anthropomorphized "AI" as if they're super-human intelligent entities capable of conversing as humans do, but also better. And when they inevitably fail to live up to that, the narrative shifts to the need for "alignment" and "safety", which seems to mostly mean adding some words to the short priming prompt to make it avoid outputting some specific type of thing that was found to be problematic (with the definition of problematic generally being the creation a public relations shitstorm while also not creating profit). But us humans, our only benchmark for comparison, develop our sense of wisdom much slower through repeated learning and development stages. You can't simply read a child an essay of any length about how to drive safely and then hand them the keys to the family car.
As always, HN is a very biased audience. Most of us have probably read about how LLMs are the best bullshitters in the world, and we've seen a lot examples like this that prove how little LLMs can be trusted.
But in the general population? I'd bet there's plenty of intelligent people who haven't had the same exposure to LLM critiques and might trust their output out of ignorance rather than lack of intelligence.
After all, people with higher IQ are used to reading books and believing them, whereas those with supposedly lower IQ tend to dismiss books and experts, instead believing in their own real-world experiences.
I'll provide an example from a comment I wrote a while ago: https://news.ycombinator.com/item?id=35540356
Some person with children who dies on an idiotic way has done more for evolution than the still alive and childless me.
They do account for future ability to reproduce, whether the person dies or not. For example, https://darwinawards.com/darwin/darwin2000-38.html
The bottle being labeled is sufficient, just as the LLM being labeled as sometimes wrong is sufficient.
Labeling LLMs as "sometime wrong" is like labeling drugs as "sometimes have side effects, especially when mixing". It's a truism, such label would be completely useless. You need to take the drug anyway, so knowing that "some side effects exist" doesn't change anything. And often you also do need to take 2-3-4 drugs at same time, so the mixing clause is not helping either.
It took us many decades to build the system around drugs that today you take as granted. Drugs are studied very carefully, with all observed side effects listed explicitly. Drugs compatibility is a big topic and all new drugs are checked in combination with common-use drugs.
At the other end of equation awareness of side effects and mixing was increased dramatically both among doctors and patients, who were previously completely oblivious of them. Mere 100 years ago people were prescribed heroin against cough. Only 60 years ago the thalidomide disaster happened.
If all you can say to people destroyed by Kevadon is "you are at fault, the bottle said CAUTION: new drug", then I'm afraid we see this too differently.
English is such a fun language sometimes! So many idioms.
I suspect if you haven't experienced how effective LLMs are at making coherent prose that is nevertheless utterly incorrect, you either haven't been asking very technical questions or you've been quite lucky.