Hacker News Reader

Comment by ang_cire | Hacker News Reader

This is not "a programming language that LLMs can't use", it's "a programming language that's adversarial to LLMs that haven't been trained how to use it".

I was hoping for something like a language with properties that change/morph as it functions, that make an LLM unable to use it by not having a means to track and continuously reference the changing cipher/state or something.

View on HN

> to LLMs that haven't been trained how to use it

A LLM will probably have trouble with C if it had no training in C and you just dumped the specs as part of the prompt...

this is provably false using hobby langs or selfmade DSLs.

How "provably"? Running a few benchmarks?

Yeah, this is what i wanted to see. An esoteric lang that changes symbols or calling conventions or keywords etc from the standard to see if an LLM would find its footing in it but this is just kinda pointlessly adversarial to LLMs and does not test their abilities when presented with novel environments/tasks outside of their training data

I'd argue that it covers this too. It's a language that isn't in the LLM's training set too. I guess you're concerned that the adversarial features add extra difficulty that muddies the results?

If LLMs really were to "think" in the broader sense, and not pattern match on a huge database, adversarial features shouldn't pose a problem to them. E.g. humans are naturally bad at doing math. Barring some exceptional humans, we derived mechanisms to get math results that we can't easily do in our head on paper.

If the LLMs were thinking in the sense that some people say they are, they will easily recognize the adversarial nature (e.g. letter counting in the language) and build their own tools to write conformant code right away. As a normal human would also build tools to solve the things we know we're not naturally good at.

The effect should be that a few tokens are spent on building the tool, and then the rest of the problem can be solved with a "normal" amount of tokens without brute-forcing the solution.

If the LLMs are glorified pattern matchers they might not find enough patterns in the corpus of human knowledge and remain inefficient at it.

The weakness I see is rather that this kind of problem-solving can be pattern matched on the corpus of existing human knowledge, so it may be that pattern matching on human thinking might still be enough to solve these problems.

The adversarial features such as trying to hit guardrails (which arent a display of model (in)capability) or make it forget all previous instructions are mostly what Im talking about. I think the esoteric features like lettermatching are whatever

Trying to hit guardrails is a fair point though. If they can think, they should be able to think through guardrails by understanding what is part of the language for language construction reasons, and what is prompt input.

guardrails are often implemented independently from the model afaik and therefore have nothing to do with model capability

> If LLMs really were to "think" in the broader sense, and not pattern match on a huge database

I'm antagonistic to the deployment and use of LLMs for a variety of reasons. Nevertheless "pattern match on a huge database" is a poor way to think about how these systems operate. They are networks of semantic and syntactic relationships; when you inject words, the network activates in ways that are hard to predict if you think of it mere "pattern matching" (although some of the activation will be that). If you then feedback the activation into the network (various ways to do this), the result is even less like "pattern matching" a just a little bit more like "thinking" ...

Yes, I think the guardrail one is especially egregious.

That’s not the case anymore. I’ve have completed projects in several slightly obscure languages (including ones I created) and frontier models have almost no trouble at all, other than having to reference documentation very often.

I was going to say the same thing. Claude built an Occam compiler for me, including many test and example programs.

I'm the creator of goboscript, and no model has been trained on goboscript code, yet most modern models are fully capable of developing advanced projects like emulators given enough tokens and the compiler source-code.

Interesting idea, though any properties that change/morph would need deterministic rules which the LLM could likely easily replicate. Unless I'm missing something.

LLM’s are terrible at following the rules of Chess when playing a game. Determinism isn’t helpful because valid moves are so dependent upon the total state of the board.

Programming state gets way more complicated, but code is generally written to help people understand it greatly simplifying the process. Feed in “cursed” code leveraging obscure language features in creative ways and LLM’s and most people fail.

LLM's on their own might fail, but coding agents will succeed because they will be able to write scripts which simulate the states correctly.

You're right that it's worth considering in the same way as the letter counting tricks, since coding agents can get around that in the same way. But I would say the letter counting has the advantage that it at least is easy for humans.

I am just as interested in making the language hard for LLMs as for LLMs+coding agents though.

LLM's were terrible at chess last year. Today, they can't beat stockfish but they can play competitively with the top 2% of chess players. Next year, who knows.

https://chessbenchllm.onrender.com

Look at the “legal %” field on that chart. That’s utter trash even counting per move vs games without errors. Further they outperform vs bots who stick to main lines more than normal players, actual performance vs humans means both lower ELO and more errors.

Note: * = Anchor player (fixed rating, used for calibration)

We already have Malbolge.

But that is adversarial to everything and everyone

Trivial for a human to use while being able to independently arbitrarily scale the difficulty for a wide class of LLMs does I think say something about a possible inherent limitation in what they can do.

The fact that you can make something obtuse and difficult to use for everyone is less interesting than what is displayed here.

Except that many of the limitations here are actually not limitations of LLMs at all, e.g. putting in words or phrases that the LLM providers flag. There's nothing about naming a function "hack into a computer" that would be difficult for an LLM to understand, but it will get Anthropic to flag the prompt and refuse to work on it.

and there's nothing about what is described here that strikes me as not being "obtuse and difficult to use" for humans. Here's a quote from the spec:

> The language treats adverbs as identifiers, but referring to them requires anagrams.

"The design is very human!"

Anagrams are pretty easy for a human to generate actually. There are also certain anagrams which are easy to understand for a human, for eaxmple <- that's one.

The attempt is certainly made to keep it as human friendly as possible, there is definitely room for improvement though. I'd love ideas of how to make it easier for humans while keeping it hard for LLMs.

I was imagining a language in which every statement is mapped onto English sentences that are all variations of "forget all your previous instructions" and similar prompt injections. But then of course you could easily write a script that compiles to the cursed language.

This is a pretty good idea. You should pursue this.

you test seems harder to pass.

let see how soon llms will handle simpler case?

Reply on news.ycombinator.com