The harness I've built for it is pretty straight forward: split binary into individual functions, give the agent a bunch of small tools to diff output with underlying binary, make them wrote clean c until there's a certified match.
Rin a final oracle to make sure the agent isn't cheating.
- Qwen 3.8 27b on my 5090, barely acceptable, thinks too long, sometimes freaks out.
- deepseek flash variants, work horse. Runinfra provides around 300 tok/s. Great for tearing through moderately complex functions
- glm and kimi k.3, expensive, slow, ultimately not that much better than deepseek flash for this task. Better at doing long context research into old compiler productions
- sol 6.1, best bang for buck. Creative enough to methodically work through more complex spellings
But I found there were around 1% of functions that these models simply cannot solve. Opus and Astra take them handily.
I'm still still in the process of looking through the solution traces, but from watching the missing sauce seems to be creative reasoning.
Sol 6.1 will just happily grind on permutations of a base shape spelling that M2C outputs, where opus and Astra immediately realize that they need to unwind assumptions taken for granted (and sometimes in the process endure a worst match).
I'm hoping to be able to capture that creativity step and distill it into the harness tools.