We already know the trick to get massively improved coding performance from an existing model: set up a mechanism for it to execute the code that it writes and then react to any errors or test failures that occur.

ChatGPT Code Interpreter has been demonstrating the effectiveness of this trick for over a year now. Anthropic are working on their own version.

I got Code Interpreter to write, compile, debug and fix C code the other day, despite it pretending it can only work with Python: https://simonwillison.net/2024/Mar/23/building-c-extensions-...