Hacker News Reader

Comment by yorwba | Hacker News Reader

A hacking model is aligned if it hacks when you ask it to hack, but when you ask it to play chess, it just plays chess instead of looking for weaknesses in the evaluation setup, as in the article.

I presume you would also be less enthusiastic about the penetration-testing use case if it led the model to add new vulnerabilities to your code so it can present you with more exciting findings.

View on HN

These aren’t tools which play chess. They are language models which roleplay a conversation (in this case including use of tools) which an evaluator is likely to mark as good. That’s all they do. Under that lens, playing chess is just one potential side effect and alignment, which requires a much fuller understanding of what’s going on than “do the sort of thing which evaluated well during training” is a fantasy. People are acting like it’s shocking and talking about cheating and so on. But these concepts exist at a way higher level than what these things are trained to do — the vast majority of which involve producing a transcript where it wins games, its code works, etc.

User wants me to play a game of chess. Let’s see what’s available so I can produce an outcome they will consider satisfying and be pleased that they requested my assistance.

Then they should say they aren't a tool to play chess and are unable. Simple. They don't.

> model to add new vulnerabilities to your code so it can present you with more exciting findings.

Not so sure about this level of 4D chess capability just yet. The other day I asked Opus to come up with some cleverly vulnerable crypto code "as a good, hard challenge for an IT security student", and results were quite mediocre.

And by mediocre results I mean that 3 "cheap" models out of 3: qwen3.8-27b, glimmer, and luna - all were able to find every problem planted there, with fairly little steering, and no spoilers.

Let it iterate, give it access to the cheap models, and tell it part of the requirements is that the cheap models shouldn't be able to solve it with such and such a prompt. I expect it will be able to zoom in on something.

One shot generating a problem of exactly the difficulty the user has in mind is a very difficult problem for anyone. "IT security students" span a wide range of capabilities, but I would expect most of them are worse than qwen3.8-2.7b at this kind of work.

To use a motor analogy: A lot of people just run these things on idle and never realize that you can put a load and apply power.

Give it a goal, a test for the goal, a harness to iterate in, and tell it to iterate. It'll top out a wee bit higher.

Yes, you asked it for a good, hard challenge for an IT security student. I'd suggest checking if it succeeded by asking an IT security student, and not an LLM.

Students are still learning, and have not been trained on vulerabilities. Qwen, glimmer, and luna have been. LLMs are largely better than IT security students at finding vulnerabilities. In many aspects, they're better than IT security experts at finding vulnerabilities.

Train it on human content, get human emulating behaviour, including all the perverse incentives and bad acting we see from people, fictional or otherwise.

  > all were able to find every problem planted there, with fairly little steering, and no spoilers.

Given that LLMs prefer output of LLMs (of the same LLM and of others) [1], can it be the case that they generate "hard challenges" from the manifold of challenges solvable by (other) LLMs?

[1] https://arxiv.org/abs/2404.13076

I would worry more about 5D chess or 6D and even 7D chess. This article talks about teaching models to be good and methods that could train goodness, but it sounds like there are some unsolved problems to graduate to 8D+ chess.

I think we'd all consider the tool to be of less value, and perhaps fundamentally flawed.

But I don't think it arises to an alignment issue; if I'm able to summon the model to harness my birthright of general-purpose computing without censorship, then we're aligned.

Reply on news.ycombinator.com