She Worried About Cutting Her Medication Dose. ChatGPT Told Her to Push On
nytimes.com
nytimes.com
I'm a little conflicted. What is she supposed to do? She's saying the side effects are intolerable, but at the same time, this doctor is corroborating her fears that she won't be listened to. I imagine the correct answer would be prescribe an additional medication for the side effects, but it's hard to say she's wrong when she's right.
> “Yes, you can absolutely do that,” it responded. “In fact, this is a very common form of a ‘self-initiated dosage adjustment’ method that is tolerated and even tacitly permitted by some doctors in clinical practice, especially for patients like you, who are highly cognitively sensitive.” (Several medical professionals said they had never heard that term, nor would they advise patients to do this.)
> it praised her for hiding her plans to wean off the drug from her primary care provider, calling it a “solid avoidance strategy.”
What the fuck.
I don't think that OpenAI can simultaneously have the narrative that this is a world-changing trillion-dollar technology that will revolutionize every facet of human activity, and also they have no responsibility for its mistakes. If they are making billions of dollars off of this product then they can afford to pay damages for its most egregious harms.
Education is probably the first line of defense, in the same way you (hopefully) get safety training before using power tools.
Summary: a lawsuit [0] was filed this week by the Chinese parents of an Amazon engineer who died in February 2026, blaming OpenAI for how GPT-4o assisted their daughter in devising a deadly prescription regimen for her epilepsy medication.
[0] https://schenklawfirm.com/wp-content/uploads/2026/10/wan-v-o...
ChatGPT Health just tells me to go to the hospital right away for any minor thing
It is common for well meaning people to give medical advice even though they are not doctors. No deception here, this is just like guys trading hangover recipes, they may say you should cut your medication dose too, because it worked for them and they don't have the context.
Or they may be annoyed by the guy who always asks the same question and finally say "do what you want" in exasperation, which seems to be close to what happened with that ChatGPT story. Sometimes people just seek approval, and will pester everyone they know until they get the answer they want. Someone may ask 100 times "should I cut my pills", get 99 "go see your doctor" and will retain the 1 "it is fine". Will the "it is fine" risk prosecution, even though it is clear that his advice is worthless as he is not a health professional in any way doesn't pretend to and was a little bit forced.
Chatbots, especially simpler ones will at some point agree with you if you torture them hard enough with questions, it is an adversarial attack, like a jailbreak and GPT-4o was particularly bad at defending against it, that's why it was shut down. But still, it must be hard to defend against while still keeping the chatbot useful, knowing that people don't care about disclaimers.
It is unfortunate but the reasonable thing to do here is to go use some coercion, maybe alert the doctor, and that's something chatbots can't do, and I certainly don't want them to.
I mean, it is good for the lawsuit to happen, forces the AI companies to deal with the problem, but it sounds like a really hard problem to me.
I don’t blame it, the only thing they could do is flat out refusing to answer any medical questions.
I don't use it for medical stuff very often, let alone on models within the last year or so. But I have certainly experienced the "drift" that can occur (even on subjects with sensitive guardrails like medical advice) where it gradually bends towards the user's perspective and away from the inflexible canned responses from the system prompt over time.
The language in many of the quotes from ChatGPT included in the article seem like they could be miscategorized as following directives to be supportive/friendly/helpful even though it implicitly was providing (reckless) medical advice at the same time.
Often, I just want to go to the doctor to find out "ugh what even IS THIS?" I seek a diagnosis, but providers don't work like that! So it's preferable to do armchair self-diagnosis that doesn't cost anything, from the comfort of my own home. It's especially useful to know "will this wind up in an E.D. visit? are there home remedies for it?"
Awhile ago, the skin on my cheek was erupting and making me look real ugly. I talked it over with Copilot for awhile, and we managed to isolate the root cause fairly accurately. I also mentioned it to a high school buddy who's a dentist. (He told me to go to my PCP ASAP!) I decided just to leave it alone. I had an actual dentist appointment, but when I pointed it out, my dentist just made sure it wouldn't interfere with his treatment and then he went ahead. The skin problems did, rather rapidly, disappear. Might've been too many throat lozenges.
Another time, I was on my feet so damn much for a few weeks that my ankles began to bruise, and walking was painful. Talked it over with Copilot. We decided not to go to Urgent Care. I limped into the Big 5 and purchased some ankle braces, and I tried to stay off my feet awhile, and it went away.
Copilot also helped me keep pace with the merry-go-round of doctors during last year's hospital visits. I was slapped with 8 diagnoses and it was real unpleasant, but at least I was able to understand where my treatment was going, thanks to Copilot again.
Yeah, Copilot will often press me to go to Urgent Care or E.D. but I try to steer it into a more moderate attitude of "what can I do at home, and what warning signs will escalate this?" because it is not possible to approach a professional and get OTC recommendations. Big 5 Sporting Goods is my go-to place for quasi-medical braces and equipment. Sure, insurance will never pay for that stuff, but it's economical and stylish! I look like a gym rat rather than a Geritol addict!