dom96

So far only like this

I wish I could benchmark Muse Wonder Full, because Muse Wonder isn't much better than Muse Spark 1.3[1]. I created a brand new programming language designed to confuse LLMs[2] and created a model benchmark out of it.

A lot of the tasks were failed because they exceeded the $5 budget though, so maybe it would have gone on to complete them, it just required a lot of funds to do so. Still, that's the benchmark, and it's not better than MiMo V2.6 Pro which is pretty disappointing.

1 - https://bench.killswitch-lang.org/

2 - https://killswitch-lang.org/

I created a brand new programming language to test LLM models on[2]. It looks like Mistral Large 4 fares about as well as Kimi K3 and Claude Sonnet 5.5[1]. Though it is much slower than Sonnet 5.5 and more expensive than both of those.

1 - https://bench.killswitch-lang.org

2 - https://killswitch-lang.org

Seems to be worse than MiMo v2.6 Pro in almost all aspects, in particular on cost, but also intelligence. Kinda disappointing.

Excited to test this on my benchmark[1], but I'm guessing it won't fare better than MiMo v2.6 Pro which is currently the best open-weight model as far as I'm concerned.

Why isn't this on openrouter yet? Is there a better router that gets models much quicker?

1 - https://bench.killswitch-lang.org/

Matter/Thread isn’t proprietary, it’s an open standard

I've been thinking about installing a ceiling fan in my UK victorian home. Those in the US may not be aware, but ceiling fans are really not common in the UK. So it's been rather difficult to find one.

But really what annoys me the most is that they seem stuck in the 90s. Even the really expensive ones come with crappy white remote controls. Why aren't they just coming with a Matter/Thread chip so I can control them from my Apple Home?

Why aren’t the data centres just honest about their water usage if it’s seemingly so low?

There is a reason I don't use Arch though lol

I’ve tried Pi and honestly it felt pretty rough and raw. Kind of like Arch Linux. Even with oh-my-pi the experience just didn’t seem polished enough for my taste. I’m not really sure why I would use it over OpenCode.

I built a brand new language to test this[1]. Not only is the language different to basically any other language but it also tries to be adversarial against LLM understanding.

The best models can still make sense of it[2], though the tasks so far have been pretty basic. But I do think it gives some evidence that languages which aren’t well represented in an LLM’s training can still be reasoned about and written well by LLMs.

1 - https://killswitch-lang.org

2 - https://bench.killswitch-lang.org

I'd love to run it on my benchmark but alas, Google not making it public prevents this.

Why announce this if it’s not available yet? Why not at least announce when it will be released to the public?

None of the other AI labs do this. Really frustrating.

Well, it is genuine.

Opus 5.5 fails the "understanding" tasks which Opus 5 passes. I feed it a script which takes two numbers and prints the max of the two numbers. Opus 5.5 thinks it prints 1/0 instead of the max numbers. Opus 5 gets it right.

Here are the outputs from both: https://gist.github.com/dom96/b5bce82b6e6c1ebd5271ed70ad941b....

Looking at that Opus 5.5 fails to deduce that the "hack statement" is actually an if statement in disguise, but Opus 5 gets this right. I feel like this is a pretty good test and shows Opus 5's greater intelligence.

Based on my benchmark[1] it is the same price as Opus 5.5 and just as capable.

1 - https://bench.killswitch-lang.org

Surprisingly (or maybe not) it matches the performance of Astra on my benchmark[1], but is much cheaper. It is also head to head with Opus 5.5 on both the price and pass rate, but edges it out slightly.

1 - https://bench.killswitch-lang.org/

Funnily enough the Fable safeguards are the worst and testing Sonnet 5.5 didn't trigger them as much as it even did for Opus on my benchmarks[1].

1 - https://bench.killswitch-lang.org

I built an adversarial esoteric programming language to benchmark LLM models and just ran it on Sonnet 5.5 It does worse than Sonnet 5. Mainly because it is more reluctant to keep going to get an answer, instead it returns to ask the user questions whether to keep going.

https://bench.killswitch-lang.org/

    Claude Sonnet 5    17.8%
    Claude Sonnet 5.5  7.4%

There are multiple phrases that can be used[1]

1 - https://github.com/dom96/KillSwitch/blob/main/SPEC.md#abrupt...

Agents are fairly good even at languages designed to trick them. I built one[1] and it does make for a good benchmark[2] to see which LLMs are actually good. I think that a language which is largely similar to others will be a piece of cake for most and any advantage that an existing language will have will be minor enough to not matter.

Btw if folks have ideas of how to make Killswitch even harder for LLMs I’d appreciate them.

1 - https://killswitch-lang.org

2 - https://bench.killswitch-lang.org

The Democrats are not a left-wing party, I'm not sure why you think "the Left" would be in love with him after all the other horrible things he's done if he just gave money to a political party that is at best centrist.

If you want him to get the benefit of the doubt about his "hand gesture" then it would help if he wasn't promoting far-right parties all over Europe.

It's surprising but MiMo V2.6 Pro performs better and is cheaper than GPT 6 Sol on my benchmark[1]. Open weight models are really snapping at the heels of the major western models.

1 - https://bench.killswitch-lang.org

Just updated KillSwitch-Bench with this new model: https://bench.killswitch-lang.org/

It does perform slightly worse than Opus 5, but it is significantly cheaper and faster.

Not yet. But can do so if there is interest.

I’m using openrouter which I think is a fair representation of what the typical user will experience.

It is an impressive model. Agreed on most that is written on this page, with the exception of it being fast. I ran it on my own LLM benchmark suite[1] and it is faster than DeepSeek but still much slower than leading models. But it's pricing is where it really shines.

KillSwitch-Bench 1.0

  Claude Opus 5           66.9
  GPT-6 Astra             57.9
  Claude Fable 5.1        46.7
  MiMo-V2.6-Pro           38.8
  Muse Spark 1.3          36.5

1 - https://bench.killswitch-lang.org/

How do you think other benchmarks work? Every single one is going to have a budget and time limit. There has to be a cap on those, you can't just let it spin forever and use unlimited funds.

I would argue that this benchmark is uniquely suited to how most people use LLMs because it actually tests common harnesses and it is a true coding benchmark for a language that is unseen, thus testing the LLMs actual ability to understand nuance and learn.

Internet access is restricted to ensure that over time models cannot just look up the source code of KillSwitch, which would allow them to cheat.

Very capable model. I just ran it on my own LLM benchmark suite[1] and it matches Muse Spark 1.3 in pass rate but is significantly cheaper.

KillSwitch-Bench 1.0

  Claude Opus 5           66.9
  GPT-6 Astra             57.9
  Claude Fable 5.1        46.7
  MiMo-V2.6-Pro           38.8
  Muse Spark 1.3          36.5

1 - https://bench.killswitch-lang.org/

Why not? In my own benchmark Opus 5 does in fact come out on top[1]

1 - https://bench.killswitch-lang.org/

(I'm one of the authors of this post)

> Being tied to use only one version of Python/Pyodide (the one that Workerd embeds)

This isn't quite the case, you can choose between different versions using compatibility flags. For example, `python_workers_314` is the compat flag for Python 3.14[1]. You've also got compat flags for 3.13 and 3.12. Though it is worth noting that by using those older versions you will also be using older Pyodide versions too, which have fewer features (for example they lack JSPI support).

> Architecturally tied to the JS/v8 world, which may show some challenges as they aim to reduce cold start times

That is indeed a challenge. But our memory snapshot implementation has improved the cold starts significantly already and we will be working to reduce these even further. We also have sharding these days which reduces cold start frequency a lot. We wrote about cold starts (and sharding) in a previous blog post[2] which includes some numbers.

1 - https://developers.cloudflare.com/workers/configuration/comp...

2 - https://blog.cloudflare.com/python-workers-advancements/