The problem is that this approach relies on a large language model trained on a copy of the entire web (GPT-3, trained on Common Crawl plus network extras), fine-tuned on github (Codex) and then fine-tuned again on the kinds of problems that it is supposed to solve. That is an insane amount of resources to spend to solve simple programming problems with known solutions that can be implemented by hand at much lower cost and effort. In that sense it's a little bit disappointing: so much data, so much compute, and all you can do is tell computer how to code python?