[flagged]
- Open source models found the same bugs? Sure, if you tell them "here is a file which may contain a vulnerability, look for a bug in how function XYZ handles ABC"
- It's all mostly false positives? According to Anth, each suspected vulnerability came with a bug report and working PoC...
- "Humans had to fix the things"? As in, he thinks models are incapable of writing the patch?
In one of Anthropic's blog post, they describe that that's basically what they did too. They run the agent many times, each time specifying a different file to focus on. [1]
From my experience as a security researcher, manually finding a fishy file and sicking even sonnet 4.5 yields great results for most memory corruption bugs.
No comments otherwise. I don't have a clue as to who that guy is, and I haven't watched the video yet. You might be right overall.
[1] https://red.anthropic.com/2026/mythos-preview/
Edit: looked at the open source model claims - I agree that they suck. Basically all the details are given away in the prompt - not just the file.
If OSS models are equally up to the task, why not find novel vulnerabilities?
> If OSS models are equally up to the task, why not find novel vulnerabilities?
To be fair, in the same blog post Anthropic mentioned costs in the tens of thousands of dollars per project looked at it. So it's a big ask to do an experiment that compares. Would love to see it though.