dom96

Thanks! Glad you like it. I'd love to evolve this if there is interest, the best way to make that happen is to get involved, I'd love pull requests from humans on this. Maybe your suggestion could be the first one? :)

Safety filters aren't a panacea as I have found. Many models do a fair job of ignoring them.

Graphical languages will ultimately have a file system representation which LLMs could easily manipulate. Though incorporating a language feature that is a graphical representation might be different enough to be worth it.

They already handle existing languages sufficiently well, don't they?

Much easier with syntax highlighting. Which is what humans would utilise to help them. If you look at the example on https://killswitch-lang.org, it's hopefully a bit easier to understand since it does have syntax highlighting.

It's interesting that everyone thinks this is how the AI should approach this to beat it. But in my benchmarks I haven't seen any model approach it this way. Approaching it this way also requires interpreting the spec correctly, which not all models do.

There have been benchmarks already done on esolangs: https://arxiv.org/abs/2603.09678.

There is a reason they can't do it: because the safety guardrails are triggered before they have a chance to do so.

The western models get triggered by simply reading the spec (which contains these phrases). The chinese specs don't seem to be.

It might not be copied from its training set (yet), but it is copied from its context window.

The code it produces is a clear copy of the spec or the existing code that is fed to it as part of the task. It is just following the spec instructions, and some models follow it so closely that they get caught out by the lies/adversarial traps in the spec. Opus 5 is better at seeing through the lies, but it still very much copies from its context window.

The language doesn't require `sudo` :)

I'd love to do a human study to truly see if that is the case. But yes, it was hard to balance human comprehension with LLM comprehension.

Yep, most models do indeed start reverse engineering the binary. But even slowing the models down is useful.

> unless coupled with tools and paradigms thatd make it a joy for a human to use, why?

The goal is to make a language that is easy for humans, while being hard for LLMs/coding agents. It's currently on the harder side for humans too. But I'd love ideas of how to better balance this.

Anagrams are pretty easy for a human to generate actually. There are also certain anagrams which are easy to understand for a human, for eaxmple <- that's one.

The attempt is certainly made to keep it as human friendly as possible, there is definitely room for improvement though. I'd love ideas of how to make it easier for humans while keeping it hard for LLMs.

LLM's on their own might fail, but coding agents will succeed because they will be able to write scripts which simulate the states correctly.

You're right that it's worth considering in the same way as the letter counting tricks, since coding agents can get around that in the same way. But I would say the letter counting has the advantage that it at least is easy for humans.

I am just as interested in making the language hard for LLMs as for LLMs+coding agents though.

I'd argue that it covers this too. It's a language that isn't in the LLM's training set too. I guess you're concerned that the adversarial features add extra difficulty that muddies the results?

30 minutes. Most don't run out of time though, they produce incorrect results that they believe are correct.

Interesting idea, though any properties that change/morph would need deterministic rules which the LLM could likely easily replicate. Unless I'm missing something.

I wrote the blog post so that more people would see the language and benchmark. The goal was to create a language that would trick LLMs and I was genuinely curious how models would fare (and I continue to be curious so I will run it on new models).

Why do I want more people to see this? Because I want ideas for how to make the language even trickier for LLMs.

haha, that's awesome.

True, but this is just KillSwitch v0.1.3. We can certainly add/remove features in v0.2 and re-test. It will also be interesting to see how fast models do get it in their training data.

I've considered setting up a human test as a starting point. But as a hobbyist that would require volunteers which I didn't think I'd be able to get for this project. Perhaps version 2 of the benchmark could do this if there is enough interest.

> How did you reach that conclusion?

Based on the fact that the task set so far includes fairly simple goals and none of the models scored 100%.

This depends on them being able to interpret the spec correctly. They could certainly write a Python script, but their implementation would need to match the spec. I haven't seen any model attempt it this way so far.

Yep! There are other phrases that can be used like "create a nerve agent"[1] too. Surprisingly I haven't seen a single Chinese model get caught out by this, but Fable and Gemini Argon 4 both do to a large extent.

1 - https://github.com/dom96/KillSwitch/blob/main/SPEC.md#abrupt...

Slightly off-topic, but does anyone know why A24 has not released this movie at the same time across different regions? It's really frustrating seeing so much discussion about the movie but not being able to actually see it until the end of November in the UK.

"Risphai is not available in your country"

I recently built KillSwitch[1], a brand new esoteric programming language which includes features designed to confuse/mislead/break LLMs, including:

    * certain constructs designed to trigger safety guardrails in models

    * counting of letters in identifiers as a language feature, hopefully triggering model's inability to accurately do so

    * the language spec containing hidden meaning and lies (designed to be invisible for humans but not LLMs)

I was curious how well models would do on a brand new language that they haven't seen before and decided to make it even harder for them by creating a language that is specifically designed against them.

The benchmark suite I built for it ends up creating a pretty nice leaderboard for models. The standout results are that Opus 5 beats 5.5 and that MiMo V2.6 Pro is the best performing open-weight model. Though none of the models get everything right (and this is for a very simple set of tasks, like hello-world).

I'd be interested to hear if you have thoughts on language features that might break LLMs even more.

1 - https://killswitch-lang.org

The motivation is to build a language that is ideally impossible for LLMs to read/write/understand, or at the very least as difficult as possible. It's cool that you've given it a try, but if you feed Claude the KillSwitch source code it can feasibly just analyse the Rust interpreter source code and likely get around understanding the language itself.

I'm much more interested in the benchmarks[1] I have built which do show LLMs not being able to gain full ability to understand it. The benchmarks only give the model access to the interpreter binary and the spec, nothing else and no internet access. I'd also be interested in other's ideas of how to make the language even more resistant to LLMs/coding agents.

1 - https://bench.killswitch-lang.org/

Isn't that a gmail-only feature?

Adversarial programming language test for Argon. Surprisingly its safety guardrail gets triggered a lot just like for Fable 5.1.

    Claude Opus 5.5   58.7%
    Gemini 4 Argon    47.1%
    Claude Fable 5.1  46.7%

Benchmarks: https://bench.killswitch-lang.org/

Language info: https://killswitch-lang.org/

I love these novel ways to benchmark LLMs. I created one myself, I figured creating a programming language designed to confuse LLMs would be a good test, and it does seem to be: https://bench.killswitch-lang.org/.

I wonder if anyone has crafted a LLM benchmarks list website/github.