Hacker News Reader

Comment by saghm | Hacker News Reader

I don't understand what this means. I use LLMs daily for my work in programming things, and they regularly will assert things that are not accurate.

View on HN

A lot like humans, really. People regularly cite something they read, or quote a stat that turns out to be just completely inaccurate. But if you look up the thing, then you have facts again.

Sure, and by the same logic, if I was trying to cook a steak, I would not trust an arbitrary human to know the correct temperature off the top of their head; I'd want someone who I could trust had actual experience with the task I was trying to perform. The difference is that most humans are fully able to recognize whether they've cooked steak often enough to know the correct temperature off the top of their head, and they will say "I don't know" to most random arbitrary questions you ask them outside of their experience. I've yet to see an LLM product aimed at general usage for individuals be willing to say this without someone having to literally direct them to give that as an answer if they're not sure.

His point was that, just like humans, if you ask the LLM to look it up, it will give you the correct answer. Just like humans, if you don't ask them to look it up, you don't know what you're going to get.

And my point is that I've never met a human who confidently asserts incorrect information in such a broad range of domains rather than just admitting that they don't know

The point is that it's trivial to fix the problem of cooking the waterfowl or writing your code with existing tools. Just ask them to verify.

If you have to say "don't make up something" for every possible question you ask in order for it not to make up something, that's a massive usability issue for regular people. If saying "don't make up something" will still result in it making up something up some of the time, that also might be a massive usability issue depending on if "some of the time" means 0.0001% or 1%.

Having a natural language interface where you need to go out of your way to specify that you want an accurate answer rather than just a plausible one defeats the entire purpose of it being a natural language interface for normal people. In certain professional contexts, it can be useful, but I don't buy it at all that it makes sense to ask everyone in their everyday lives to go out of their way to specify that they actually want correct answers to their questions.

> If you have to say "don't make up something" for every possible question you ask

You don't. It goes in the system prompt.

In practice it doesn't though, for any actual products on the market today.

Put it in your own system prompt. They all provide tools for doing this. OpenaAI Custom GPTs, Gemini Gems, Claude Projects, etc. Or you can easily roll your own using their APIs.

When you let the the agent do a test, or tell it to read that doc first, you will ground it in reality. Doesn't mean they are 100% reliable. But without and on their own without access to grounding information, they halluzinate wildly.

The parent comment said that these types of behaviors are inherent to how LLMs work, and your response was to "ground them in facts". From what I can tell, this does not meaningfully change the original point the parent comment was making that the flaw is inherent; throwing a bunch of extra context at it to try to make it happen less often is useful, but it's still just a best-effort mitigation for the behavior, not somehow a way of literally changing the inherent nature of it.

Reply on news.ycombinator.com