Hacker News Reader

Comment by cheald | Hacker News Reader

This is going to be the big problem with the integration of LLMs into normal peoples' everyday lives; if you accept the answer as valid because it's eloquent and appears to follow some semblance of logical reasoning, you're going to be quickly misled.

I've had numerous instances where ChatGPT has produced output which looks confidently authoritative, but which I as a domain expert can recognize as wrong or even nonsensical. One needs to _heavily_ tune their Gell-Mann amnesia detector when working with LLMs, and recall that if it's not getting the things right you are an expert on, it's likely not getting the things right that you're not an expert on, either.

View on HN

This reminds me of speaking with SV investors. They’re eloquent and say things that sound plausible in the moment but then I snap out of it and realize they’ve said nothing meaningful or correct at all.

I think you'll find this in many places, I've noticed the same from SV also in DC, Berlin, etc. I think the upper echelons of society are just filled with people who often say very plausible eloquent things that aren't questioned for a variety of factors and it mostly just works for them.

Given how humans are quickly misled by sociopathic humans who are eloquent and appear to follow logical reasoning, I don't really have any hope. People on the whole are incredibly bad at that particular skill (at least in our culture).

I'm inclined to agree with your assessment, given the existence of the concept of Gell-Mann amnesia at all.

That said, I think we might have a glimmer of a chance to escape the executioner's axe here, as it were, because while a psychopath is intentionally deceptive, and will become evasive or aggressive when they sense that you're suspicious and probing of them, LLMs have no such quality, and will "happily" keep talking and exposing their shortcomings to those willing to listen.

Ultimately, though, peoples' ability to understand how appropriately to trust LLMs will rest on their ability to understand their own capacity for being too-trusting and easily misled, a feature on which the human psyche does not have a great track record.

Reply on news.ycombinator.com