And people want to seek medical device from an AI that thinks eggs can melt…
The problem is that people are actually finding it useful for all of those things!
I, too, would never have drawn up a functional specification for a product release which said the product might hallucinate and might be parrot incorrect facts about eggs melting if you phrase the question wrong[edit: if you insert an incorrect premise in the question], or might drive into a bus if it had a mural printed on the side. I'd have required correct performance to a lot more 9s before releasing the thing to the public than the products which have been released to the public demonstrate, or at least required a confidence metric that would allow the product to respond with an error message and escalate to a human if it thought it might fail (for tech support, not for self-driving cars, obviously).
And yet here we are. Turns out people and corporations do find utility in hallucination-prone, often-wrong, easily misled, easily confused LLM chatbots. And they'll strap a weight to the wheel of their Tesla so they can stare at their phone while it drives them on Autopilot.
For starters, I don’t give a shit what ChatGPT has to say about various programming topics.
Second, it makes sense that ChatGPT would get the facts of immutability utterly, childishly incorrect because fad oriented development has decided that claims absent evidence is the way to write software.
One generation earlier, the argument was “yeah well Google/Wikipedia says …”. So this is nothing new.
When I searched something, I was given lots of results and could individually make judgements. With ChatGPT, people are treating these answers as authority at a much more dangerous level than you would have from engaging with a search engine or sourced material.
I will grant that yes, googling and Wikipedia can also get things outright to subtly wrong, but the presentation can make those inaccuracies more apparent.
There have been expert systems since the 90's that perform better on average than doctors when diagnosing illness in patients, and / or help doctors make better decisions.
No they are.
>There have been expert systems since the 90's that perform better on average than doctors when diagnosing illness in patients,
No there has not been.
Context is important. 66% accuracy on cases that stumped doctors and took extensive testing is not trivial. It's not the same as 66% accuracy for everyday diagnosis.
Expert system diagnostic tools aren't that great.