People keep evaluating LLMs on essentially zero-shotting a perfect solution to a coding problem.

Once we use tools to easily iterate on code (e.g. generate, compile, test, use outcome to refine prompt) we will turbocharge LLMs coding abilities.