Hacker News Reader

My gut feeling is that the softcap for coding skills of current LLMs is somewhere here. There is only so far you can go if your model is a directed graph that cannot think/symbolically execute while it has to write code by emitting it incrementally top to bottom. A different architecture is needed to go further.

View on HN

We already know the trick to get massively improved coding performance from an existing model: set up a mechanism for it to execute the code that it writes and then react to any errors or test failures that occur.

ChatGPT Code Interpreter has been demonstrating the effectiveness of this trick for over a year now. Anthropic are working on their own version.

I got Code Interpreter to write, compile, debug and fix C code the other day, despite it pretending it can only work with Python: https://simonwillison.net/2024/Mar/23/building-c-extensions-...

Did you have any success in trying automate the whole development/debug loop similar what the auto coder tools are trying to achieve?

Code interpreter seem to have found a path to perfection, i don't care how bad the first response is (if there have to be >1 turns) as long as we can sync up quickly from any misunderstanding, mine or theirs. Here they kind of made the most easily nudgeable code LLM and won the benchmarks that way.

https://opencodeinterpreter.github.io

Reply on news.ycombinator.com