> Please explain to me, how an LLM can easily distinguish between information it knows and a guess.
Sure:
- explicitly stated in the context window or logically follows from what is -> evidenced or mechanically follows, thus not a guess
- not explicitly stated in the context window and doesn't mechanically follow from what is either -> not evidenced nor mechanically follows, thus a guess
Through the power of indeed being human, one can then cut a reasonable threshold for such guesses and degrade "never fucking guess" to "be very certain in the actions you take". This threshold-setting and gap identification then is thus a matter of alignment. You ask the model that - to the extent it has a sense of it - not take risky leaps. It's basically a reminder or a theme signal.
And there you go. In spirit, yes, this is indeed similar to "make no mistakes" [0], but I do believe it to have a bit more tangible grounding. The models will absolutely have a latent space (if not many) for representing uncertainty and likelihood, as well as for metacognitive considerations relative to uncertainties and likelihoods, since these concepts are very, very extensively represented via all kinds of literary devices throughout human literature. To be able to respond to things not working out as expected, one has to be able to depict hedges. To do so efficiently, one must be able to keep those efforts calibrated and measured. So LLMs should absolutely have a sense of their own certainty of actions, especially if they can depict them in their context window, which they totally can.
> YOU'RE HUMAN?!?!
Yes? Are you? Ever heard the phrase "playing it safe"? Surely you have a concept of its meaning? What kind of question even is this, when the rest of your comment is about arguing what an LLM can do, not a human?
[0] Have you ever remarked something supply chain worries related in a system / initial prompt, then saw your agent being indefinitely paranoid about artifact signatures and hashes? Same idea. Float the idea of mistakes, it will invite the agent to run linters and maybe spin up critic subagents. Float the idea of only ever taking low-stakes hedges, it will frequently sound how confident it is in any given action, and may prompt for your approval more often. These are memes, yes, but they're not baseless. Not sure why you seemingly think they are.