dhorthy

thanks. We have done a ton of red team pentesting with agents, both internally and through trusted partners. We're doing more continuously as we go.

yes, crazy to think that the original RPI hype was just over a year ago.

or someone could clone the SCB repo and make a fork that proves it one way or another

yep agree

they yearn to create content, generate more tokens, etc etc - agree

Yeah and I started with subsets although I did not find them representative, eg I picked 6 challenges for a subset and fable scored 33%, while on the full challenge it landed closer to 14-15% range

The part II post linked in my top comment has a list of the challenges I used for that.

To run them, just point your model at the paper, it can find the repo, etc and with a key or sub, get it running

great post - the same thing that has always been true of software engineering is still true:

Do fewer things in parallel, do them very well, optimize for getting incremental value out to users, FOCUS

It's been 30 years since the XP and Lean folks said "limit work in progress (50+ if you count the eli goldratt thing) - its still true

> AI-written emails where the commenter would rather see the prompt

this a thousand times this

LLMs are information transformers. if you're trying to take some rough idea and blow it out into something that "feels" substantive, you're just combining your high-signal idea / prompt with a bunch of meaningless bloat and noise from the model weights.

they seem to be able to do the call-stack diff stuff pretty well still

so wait is the finding that most of those skills reduce pass rates against SCB? wild

why would that worry you?

agree, i think the implication is that low quality code is harder to change in the future

somebody get this man a curl-pipe-bash stat

oh i really like the idea of flipping around the order of checkpoints and comparing results. Could be an interesting way to increase/decrease difficulty even

i will look into how easy it would be to zip up some subset of the results without leaking anything...probably doable

Yeah I would hold that models don’t know how to simplify because most rl/benchmarks doesn’t penalize complexity

Yeah the main reason I skipped fable was because we have a ZDR with anthropic and I didn’t feel like spinning up another account to circumvent that. Next run will have fable and sol

my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward

i laughed at the pelican bit its good

yes the labs will always prioritize the vibeslop dopamine casino as far as I can tell - making the models useful and addictive for unsophisticated users, sometimes at the expense or at the very least at the ignorance of the needs of power users

yeah someone will have to re-run this bench on various effort levels. unfortunately it is not cheap

I agree this is an option, and the next thing on my radar is to try with a more realistic "factory-shaped" harness where you have feedback from linters and other models after each coding episode that refines the architecture.

For readability specifically, I've found it hard to get the models to do this with prompting. If you've talked to opus/fable for a long time on prose writing you probably felt this too

> - what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is

this is a nicely succinct way to put this - a multi-dimensional space where no single metric is really useful

state space of the system is interesting too. I would guess that for any production software with dependencies like databases/third parties that might be too hard to measure, but if you can silo off parts of your system into bounded state machines, it may be a value metric on some module behind a clean interface.

I think the kubernetes control loop model is a great instance of this, a handful of scoped components that own a control loop across a well-defined state machine, that can operate / recover in the face of most network partitions or downtime - the promise of CRDTs but rather more a pragmatic approach to it

yes sol is still my daily driver for most coding tasks

I did find opus 5 quite handy for general knowledge work and visual design, without the cost of fable (e.g. the graphics in this post are made by opus 5)

but its not noticeably better than opus 4.8 in those regards, and I would not miss it if forced to go back to 4.8

no i'm spinning those up at some point this week. here's the first few prompts I used (claude opus 5 as the research orchestrator), (these were interspersed with lots of tools and assistant messages but it should get you kicked off.

> fetch this article for slopcodebench and help me run an eval on a subset of problems with opus 5 https://arxiv.org/html/2603.24755v1 > Get all the context, fetch any mentioned repos, and then propose a plan to me.

> i have an anthropic API key in .... > Let's do the three challenges with Opus 4.8 and Opus 5 and Fable please. I like your minimal set. Let's try it. What do you need from me?

> Actually I changed my mind. I want to do two of the easy ones you picked and then I want you to pick the one with more checkpoints, maybe one of the harder ones, not the very hardest one but one with a higher number of checkpoints.

> Actually let's do one easy, one medium, and one hard problem please. If we have a hard problem I'd like to see that.

> lets rock - i think lets just do opus 4.8 and sonnet 5 and opus 5 since we our ZDR will block fable

i hope that is because you hate slop and not because you write it

yeah this was just a start - the fastest cheapest thing we could try for a brand new model.

I'm hoping to do some more work with sol/fable in the mix as well as exploring more languages and curating the problem set to include more of the benchmark

I also kinda felt like opus4.5 was dumber than 4.1 personally, maybe a little biased since 4.5 was 2.5x faster and 2.5x cheaper seems to indicate its a smaller model

the one good thing about the current ios version is that it is the best it will ever be from this point forward. all future versions will be worse

I agree this rocks I do this almost daily

yes well said

don't forget the andon cord

i humbly disagree - horizontal means touching one plane of the stack across, vertical means cutting down through it and touching multiple layers https://en.wikipedia.org/wiki/Vertical_slice

i did a write up on fable while it was out - it can do big refactors, but it does not know what to change without human steering. For that, you need humans to know what to ask for.

lights off is still out for me

https://x.com/dexhorthy/status/2064747631885398231