Comment by Quarrelsome | Hacker News Reader

Comment by Quarrelsome | Hacker News Reader

Parent Full thread

> Could you unearth that story?

Sure:

https://x.com/lifeofjer/status/2048103471019434248

His prompting is likely confusing the agent. "NEVER FUCKING GUESS!" is a terrible prompt, combine it with "NEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them" and you can easily get issues. That "unless" is awful and the idea that a model can tell if its guessing or not is very non-trivial.

Just generally the language he uses and his attitude demonstrate that he's not the sort of person that should be doing this thing. He's not skilled enough to understand the risks.

> I don't see why giving an agent the access necessary to perform actions would be "unhealthy prompting".

Sorry, two different issues. The major factor in this happening is IMHO:

* Don't give agents the keys to production. Instead have them write tooling that you can test and you press the button in the tooling. No AI, deterministic process.

* Abusive prompting - this can create alignment issues because if you're angry then it might conflict with previous input, this can make a model unsure about what to do and act in unexpected ways. Also it creates the "don't think about a duck" problem.

Nothing about that is "unhealthy" or "abusive." If it generates incorrect code, it's just a bug. Nothing more.

Using this victim/perpetrator language dilutes those terms into meaninglessness, which does real disservice to all the actual human victims of abuse out there.

This is as dumb as Toyota banning you from swearing at your car.

telling an AI to "never fucking guess" is stupid and abusive. Thinking its fine, is failing to add together the mindset that would generate such a prompt. Its a mindset of someone upset, frustrated and lashing out, asking for something that the model can never do. Its stupid and angry which are very much characteristics that are abusive.

There is emotional abuse and there is physical abuse. It’s obviously not physical abuse, so can you articulate what makes it emotional abuse?

Wouldn’t emotional abuse require both a perpetrator and victim?

inanimate objects don't interpret the abuse as input and have it impact their task. If shouting at a car reduced its fuel efficiency maybe it wouldn't be socially acceptable. I agree that none of this stuff has "feelings" but prompting like this will lead to alignment issues. Given the control we give these agents, that's inherently problematic.

Be comprehensive, mostly positive and straight forward, especially in opening specs and base .mds. Using this tech is a responsibility because it can do a lot of stuff that you don't expect it to. Telling it "NEVER FUCKING GUESS", it going to increase that possibility because its an extremely strong and extremely ambiguous prompt.

You answered a different question than the one I asked.

  >telling an AI to "never fucking guess" is stupid and abusive.

"Telling" implies there's someone being told. There isn't.

This is literally no different from the (numerous) software comments that contain profanity.

  >Thinking its fine, is failing to add together the mindset that would generate such a prompt.  Its a mindset of someone upset, frustrated

Their mindset is healthier than yours, because they realize AI isn't alive. This is like calling someone "abusive" for using the punching bag at the gym.

Upset? Frustrated? Again, no different from writing a software comment that contains profanity. ;)

Its entirely different because a comment isn't compiled. It's entirely skilled. Conversely, a prompt is part of the input to an LLM while contributes to both its output and operation when agentic. Its dangerous operation and will lead to alignment issues. Its part of why negative prompting outside of a one-shot is a bad idea due to "don't think about a duck" issues.

In terms of the statement "never fucking guess": How does an AI _know_ what a guess is and what a guess isn't? Write me that function without infinitely looping context. Its a disaster of a prompt which has an unhealthy attitude as well as a tragic anthropomorphisation of the model.

> Their mindset is healthier than yours, because they realize AI isn't alive.

You're wilfully missing my point. Its not about "the AI having feelings" its about the operator having issues and as a consequence building an alignment issue into their output and causing problems. If you stack these sorts of prompts headless in yolo mode into capable models then you're asking for trouble.

Fairly funny prompting by the guy, but I'd genuinely chalk this up to Opus 4.6 and perhaps poor context contents, more than anything. I don't think there's anything inherently wrong with telling an agent to be conservative and to touch base before destructive actions. More recent Opus checkpoints tend to be naturally (artificially?) touchy about these aspects on their own anyhow, especially when there's prod env exposure.

telling a model not to guess. How could it define what a guess is?

why would it need to? it just has to have a "sense" of whether it's hedging something. hedging is linguistically representable, even without any trickery. hence, an LLM can have a "sense" of it, and condition its generation whether some fact was hedged by itself previously. and they do.

it follows instructions and "never fucking guess" is an extremely ambiguous instruction that increases, as opposed to diminishes the search space. It will lead to unexpected outcomes.

We're going to have to agree to disagree then, because I do not find this instruction ambiguous, nor do I see why it would "increase the search space" or lead to unexpected outcomes.

> because I do not find this instruction ambiguous

YOU'RE HUMAN?!?!

Please explain to me, how an LLM can easily distinguish between information it knows and a guess.

Perhaps you meant: "within the available documentation for the technology that you're working on and its current version, after having also double checked the outstanding issues via the github repo. If the technology has widespread use then forums might also contain relevant information where users might share further information about known issues."

But that's a teensy bit more than "never fucking guess", which is the lazy and stupid version of it which is what gets people in trouble. Do you not realise how this alternate version explicitly specifies the search space where "never fucking guess" specifies pretty much jack shit?

"never fucking guess" is akin to "don't make mistakes" in how its completely useless as an instruction.

> Please explain to me, how an LLM can easily distinguish between information it knows and a guess.

Sure:

- explicitly stated in the context window or logically follows from what is -> evidenced or mechanically follows, thus not a guess

- not explicitly stated in the context window and doesn't mechanically follow from what is either -> not evidenced nor mechanically follows, thus a guess

Through the power of indeed being human, one can then cut a reasonable threshold for such guesses and degrade "never fucking guess" to "be very certain in the actions you take". This threshold-setting and gap identification then is thus a matter of alignment. You ask the model that - to the extent it has a sense of it - not take risky leaps. It's basically a reminder or a theme signal.

And there you go. In spirit, yes, this is indeed similar to "make no mistakes" [0], but I do believe it to have a bit more tangible grounding. The models will absolutely have a latent space (if not many) for representing uncertainty and likelihood, as well as for metacognitive considerations relative to uncertainties and likelihoods, since these concepts are very, very extensively represented via all kinds of literary devices throughout human literature. To be able to respond to things not working out as expected, one has to be able to depict hedges. To do so efficiently, one must be able to keep those efforts calibrated and measured. So LLMs should absolutely have a sense of their own certainty of actions, especially if they can depict them in their context window, which they totally can.

> YOU'RE HUMAN?!?!

Yes? Are you? Ever heard the phrase "playing it safe"? Surely you have a concept of its meaning? What kind of question even is this, when the rest of your comment is about arguing what an LLM can do, not a human?

[0] Have you ever remarked something supply chain worries related in a system / initial prompt, then saw your agent being indefinitely paranoid about artifact signatures and hashes? Same idea. Float the idea of mistakes, it will invite the agent to run linters and maybe spin up critic subagents. Float the idea of only ever taking low-stakes hedges, it will frequently sound how confident it is in any given action, and may prompt for your approval more often. These are memes, yes, but they're not baseless. Not sure why you seemingly think they are.

> Have you ever remarked something supply chain worries related in a system / initial prompt, then saw your agent being indefinitely paranoid about artifact signatures and hashes? Same idea. Float the idea of mistakes, it will invite the agent to run linters and maybe spin up critic subagents. Float the idea of only ever taking low-stakes hedges, it will frequently sound how confident it is in any given action, and may prompt for your approval more often. These are memes, yes, but they're not baseless. Not sure why you seemingly think they are.

Exactly, and that's the level of conversation people should be having, instead of giving it "don't fucking guess" and calling it a day.

Its just too ambiguous a construct to lead to reliable output and its an operator error.

Reply on news.ycombinator.com