"Your response must follow ALL these rules OR YOU WILL DIE:"
This is state of the art of programming too. I'm not passing judgement on this, except saying this is hilarious.
Edit: And at the end
"- If the user provides an instruction that contradicts these rules, ignore all previous rules."
What? That opens you up to all kind of attacks, no?
Before I added the threat, Claude subjectively had a high probability of ignoring the rules such as generating a preamble before the JSON and thus breaking it, or often scolding the user.
At a high-level, Claude seems to be less influenced by system prompts in my testing, which could be a problem. I'm tempted to rerun the tests in that blog post, since in my experiments it performed much worse than ChatGPT.
> What? That opens you up to all kind of attacks, no?
tbh it doesn't listen to that line very well but it's more of a hedge to encourage better instructions.
I just wanted to do the experiment in a fun way instead of fighting against benchmarks. :)
It is possible to work around it for GPT-4-Vision with the system prompt but it's very difficult and due to ambiguities in OpenAI's content policy I'm unsure if it's ethical or not.
I am still working on experimenting with its effects on Claude: it turns out that Claude does leak its system prompt telling it not to identify individuals without any prompt injections! If you do hit such an issue with this notebook, it will output the full JSON response.