Hacker News Reader

Comment by SuperV1234 | Hacker News Reader

Where's the human baseline for the benchmarks?

Also, you said:

> The best performing model scores a 66.8. So does this prove LLMs can “understand” the code? I would argue that the answer to this is “no”.

How did you reach that conclusion?

View on HN

I've considered setting up a human test as a starting point. But as a hobbyist that would require volunteers which I didn't think I'd be able to get for this project. Perhaps version 2 of the benchmark could do this if there is enough interest.

> How did you reach that conclusion?

Based on the fact that the task set so far includes fairly simple goals and none of the models scored 100%.

How much time did you give them? At maximum effort I can't believe they just threw up their hands and said it was too hard. I'm guessing you really mean they failed a zero-shot?

30 minutes. Most don't run out of time though, they produce incorrect results that they believe are correct.

So they believed they had succeeded at the goal, but failed? What goal did you set? What happened instead? Did they have a mechanism to check their work?

Most tasks receive the interpreter binary which the coding agent can run to test their code (though the binary itself is adversarial too, there is a fun easter egg in there if anyone wants to give it a try). Some tasks just get the spec and a piece of code to understand. Some models interpret the code/spec incorrectly so fail that one.

Reply on news.ycombinator.com