I’m not calling to check my balance or get directions. I’m calling because your system did something wrong and needs to be overridden
And the thing is, the more the risk the less likely an LLM is going to be trusted to just make a decision.
For example, do you think an insurance agency would want an LLM to decide, on it's own, claim approvals? Can you imagine the headache for the insurance agency if the LLM approves the wrong claims or denies the wrong claims?
Or for a doctors office, Imagine AI diagnostics without a human. Can you imagine the headache for a hospital when an AI misses cancer? Or diagnoses a false cancer? It's bad enough when humans get that wrong, but now you have to explain to your legal team "We just left up the practice to the stats gods!"
You say this, but I sometimes worry that these issues are hand-waved away by decision-makers with, "oh we'll just have another LLM doing frontline claim support to verify these issues." It's the whole XML/violence thing, where the solution to XML-induced pain often ends up being more XML.
To be slightly pedantic, the Web killed travel agencies for individuals who were mostly interested in booking flights, cruises, and big city hotels once making those bookings became easy on an individual level.
Companies absolutely still use corporate travel sites, for reasons good and bad.
And there are various types of specialist tour operators, arrangers of private trips, etc. who have on-the-ground knowledge of specific locations--and often won't even book things like air travel for you.
(More broadly, the Web forced travel agents of various types to add value above and beyond what a travel portal or a random travel agent could do because they were gatekeepers to the needed systems.)
(From time to time I have some trivial questions, but not enough hundred valued bills, to ask actual doctor)
I agree with your point though, before starting on chemotherapy I would definitely shell out some money for opinion of a live doctor.