Hacker News Reader

Comment by momo5502 | Hacker News Reader

I think you're overlooking an important point here. While the token consumption might feel extremely high, I think it's a bit deceiving.

Due to this strict harness, we were able to use extremely dumb models efficiently. I think at least 200-300B of tokens (maybe even more) were processed by Luna.

Given that Astra for example is 100x more expensive per token than Luna, a high token consumption doesn't necessarily mean high cost. So these 300B Luna tokens would equate to 3B Astra tokens and roughly 15B Sol tokens.

Unfortunately a huge part of the logs is lost, so I can't do a detailed analysis on which models were used, how much they cost, etc.

View on HN

> Due to this strict harness, we were able to use extremely dumb models efficiently.

With all due respect I seriously doubt that. What are you doing that's different? Almost every test between just Opus and Opus conducting cheaper agent swarms ends up with Opus getting better results at a lower cost. People set up harnesses just like yours and see this.

> Almost every test between just Opus and Opus conducting cheaper agent swarms ends up with Opus getting better results at a lower cost.

Do you have any links to these tests? I've had a hunch that this was true for a while but haven't found any benchmarks showing it.

Interesting that you used ida mcp.

I tried giving glm 5.3 ghidra. I found that it frequently did better just reading the actual assembly, where needed writing decompilers in real time, in the terminal.

The reasoning showed that it was able to write ghidra as needed and indeed where necessary simply read assembly tokens.

Trying to think about ghidra probably confused it.

I also think that there is a gradient of losing intelligence in refusals, opposed to a simple boolean refuse. I suspect that using an abliterated model would boost the intelligence way up.

I don’t recommend always using Astra for RE but in my experience I’d use Sol as minimum. As of recently, perhaps upgraded Terra is good value, but in general it seems like you want the smartest model that is economical to run 24/7 and Astra isn’t economical.

Reply on news.ycombinator.com