Hacker News Reader

It seems possible to deliberately not train on some offensive capabilities and still have a very useful model. For example, Opus 5 deliberately did not train on exploiting vulnerabilities, and so performed less well on exploits than Mythos, yet was equally proficient at finding such vulnerabilities, according to the Opus 5 system card in their "OSS-Fuzz" eval [1].

[1]: https://www.securityweek.com/anthropics-opus-5-nears-mythos-...

View on HN

That's a good example, but I'm unsettled by Anthropic's growing refusals in the areas of chemistry and biology. If they think that scientific assistant models should be as unhelpful as Fable, because applied scientific knowledge is inherently dangerous, I don't want Anthropic or like-minded thinkers setting the standards for model safety.

It's hard to imagine a frontier model being proficient at finding vulnerabilities, but not so proficient at exploiting them.

Surely finding is the hard part, and any LLM should be able to easily exploit a vulnerability it already knows about?

No, this is not the case for human security researchers and I don't see why it should be true for LLMs.

Reply on news.ycombinator.com