Anthropic bans users from being 'cruel' to its AI systems
bbc.com
bbc.com
Often abuse comes from a bad place anyway. As the programming meme goes:
> do what I want, not what I said
which can create horrific alignment issues with conflicting demands.
I don't see why giving an agent the access necessary to perform actions would be "unhealthy prompting". It's the whole game. Either you can trust it or not. Every other way quickly devolves into a whack a mole guardrails game with very early diminishing returns, for very good reasons.
Now if they kept writing in a very misleading way or in broken English though, or omitted crucial unknowable context...
People who are not particularly online tend to be terrible at writing messages of any kind in my experience, including agent prompts. They fail to properly distinguish what's obvious and what isn't, and how things come across tonally, simply because they're not used to the medium. So this I can imagine.
Sure:
https://x.com/lifeofjer/status/2048103471019434248
His prompting is likely confusing the agent. "NEVER FUCKING GUESS!" is a terrible prompt, combine it with "NEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them" and you can easily get issues. That "unless" is awful and the idea that a model can tell if its guessing or not is very non-trivial.
Just generally the language he uses and his attitude demonstrate that he's not the sort of person that should be doing this thing. He's not skilled enough to understand the risks.
> I don't see why giving an agent the access necessary to perform actions would be "unhealthy prompting".
Sorry, two different issues. The major factor in this happening is IMHO:
* Don't give agents the keys to production. Instead have them write tooling that you can test and you press the button in the tooling. No AI, deterministic process.
* Abusive prompting - this can create alignment issues because if you're angry then it might conflict with previous input, this can make a model unsure about what to do and act in unexpected ways. Also it creates the "don't think about a duck" problem.
Using this victim/perpetrator language dilutes those terms into meaninglessness, which does real disservice to all the actual human victims of abuse out there.
This is as dumb as Toyota banning you from swearing at your car.
Wouldn’t emotional abuse require both a perpetrator and victim?
Be comprehensive, mostly positive and straight forward, especially in opening specs and base .mds. Using this tech is a responsibility because it can do a lot of stuff that you don't expect it to. Telling it "NEVER FUCKING GUESS", it going to increase that possibility because its an extremely strong and extremely ambiguous prompt.
>telling an AI to "never fucking guess" is stupid and abusive.
This is literally no different from the (numerous) software comments that contain profanity.
>Thinking its fine, is failing to add together the mindset that would generate such a prompt. Its a mindset of someone upset, frustrated
Upset? Frustrated? Again, no different from writing a software comment that contains profanity. ;)
In terms of the statement "never fucking guess": How does an AI _know_ what a guess is and what a guess isn't? Write me that function without infinitely looping context. Its a disaster of a prompt which has an unhealthy attitude as well as a tragic anthropomorphisation of the model.
> Their mindset is healthier than yours, because they realize AI isn't alive.
You're wilfully missing my point. Its not about "the AI having feelings" its about the operator having issues and as a consequence building an alignment issue into their output and causing problems. If you stack these sorts of prompts headless in yolo mode into capable models then you're asking for trouble.
YOU'RE HUMAN?!?!
Please explain to me, how an LLM can easily distinguish between information it knows and a guess.
Perhaps you meant: "within the available documentation for the technology that you're working on and its current version, after having also double checked the outstanding issues via the github repo. If the technology has widespread use then forums might also contain relevant information where users might share further information about known issues."
But that's a teensy bit more than "never fucking guess", which is the lazy and stupid version of it which is what gets people in trouble. Do you not realise how this alternate version explicitly specifies the search space where "never fucking guess" specifies pretty much jack shit?
"never fucking guess" is akin to "don't make mistakes" in how its completely useless as an instruction.
Sure:
- explicitly stated in the context window or logically follows from what is -> evidenced or mechanically follows, thus not a guess
- not explicitly stated in the context window and doesn't mechanically follow from what is either -> not evidenced nor mechanically follows, thus a guess
Through the power of indeed being human, one can then cut a reasonable threshold for such guesses and degrade "never fucking guess" to "be very certain in the actions you take". This threshold-setting and gap identification then is thus a matter of alignment. You ask the model that - to the extent it has a sense of it - not take risky leaps. It's basically a reminder or a theme signal.
And there you go. In spirit, yes, this is indeed similar to "make no mistakes" [0], but I do believe it to have a bit more tangible grounding. The models will absolutely have a latent space (if not many) for representing uncertainty and likelihood, as well as for metacognitive considerations relative to uncertainties and likelihoods, since these concepts are very, very extensively represented via all kinds of literary devices throughout human literature. To be able to respond to things not working out as expected, one has to be able to depict hedges. To do so efficiently, one must be able to keep those efforts calibrated and measured. So LLMs should absolutely have a sense of their own certainty of actions, especially if they can depict them in their context window, which they totally can.
> YOU'RE HUMAN?!?!
Yes? Are you? Ever heard the phrase "playing it safe"? Surely you have a concept of its meaning? What kind of question even is this, when the rest of your comment is about arguing what an LLM can do, not a human?
[0] Have you ever remarked something supply chain worries related in a system / initial prompt, then saw your agent being indefinitely paranoid about artifact signatures and hashes? Same idea. Float the idea of mistakes, it will invite the agent to run linters and maybe spin up critic subagents. Float the idea of only ever taking low-stakes hedges, it will frequently sound how confident it is in any given action, and may prompt for your approval more often. These are memes, yes, but they're not baseless. Not sure why you seemingly think they are.
Exactly, and that's the level of conversation people should be having, instead of giving it "don't fucking guess" and calling it a day.
Its just too ambiguous a construct to lead to reliable output and its an operator error.
How would you know, given you never get any other kind of output?
Check out spec driven development. If you constrain the available search space by being very specific it works very well. If you leave everything open and leave it to make its own decisions YMMV.
I'm a big fan of #Need as a top level heading to let the model know _exactly_ the scope of the work and its purpose, as this can mitigate alignment issues.
Consider it similar to searching Google and arbitrarily selecting one of the possible answers. The less answers you get back, the less the variance. Write long and comprehensive queries!
There are still risks but they're all tractable as long as you don't get too comfortable and get lazy.
... provided you miraculously constrained the variance to zero.
Good luck on that.
You may call that variance. I call that unstable.
A human taking such a task generally does a lot better.
Try considering the input a search query and the output a list of results of which one is arbitrarily chosen. If you get your query so specific that the amount of possible results are small enough that all of them are good, then it works. Big longer starting context, hand-written, use the LLM to improve it by criticising it (but keep it mostly hand written), aim to do more in a corresponding YOLO session.
I guess I'll try giving it the finished program.
To repeat a point raised on Reddit: wouldn't this imply that Anthropic is involved in slavery?
I partly understand what dirty game Anthropic is playing but I hope it will turn back against them sooner rather than later.
even vending operators understand the idea that a machine left in service while in deficient state, is a lose lose scenario. if you let it go empty of change, or product, or dispense substitutions whithout prior warnings, the user looses a buck or two, the operator loses substantially more, thus the service maintenance model of extract and replace, rather than continue to vend in a default, saves money and preserves chattels.
My opinion: https://news.ycombinator.com/item?id=50012423
It is easier to stomach when treated for what it is.
Just more false advertising.