'Superficially identical, riddled with bugs' is the scariest version of this. When it kept reintroducing old bugs, did writing those bugs down as tests first help, or did it just break the tests too?
Yes, it helped. In fact, it was essential. If you didn’t write a test for a bug, the model would reintroduce it over and over.
As an example, it produced PDF invoices and statements. PDFs are hard to compare, so I didn't bother. As a consequence, the stuff on the PDFs got moved, or was dropped, and the invoice didn't add up. I tried asking it to compare the PDF to a known good one - but it wouldn't / couldn't do that reliably. In the end the only solution was to ensure the PDFs it generated were always the same (so times on them had to be fixed), capture the generated PDF (not easy as it was often emailed or sent to a printer, not saved), and do a byte-for-byte compare, and make a test fail if it was different.
The fundamental difference from a human programmer is that an LLM does not learn. Once a human spots that they keep introducing errors in PDFs, they will remember and devise fixes. The LLM may do that too but then its context window overflows, everything it knows is lost, and it starts repeating all the mistakes it made yesterday. Tests are one of the few ways you can prevent it making the same mistake tomorrow.
The other issue is LLMs make far more mistakes than you realise. If you watch carefully, you'll see many of their bash scripts have syntax errors, code often fails to compile, and tests fail. Running a 2nd model over the work another produced will find many clear violations of style and coding standards. They hide this, in the same way humans hide it. They use tools and tests to find errors, like compilers and `bash -n`. They fix those errors without being asked. Compared to just a year ago, a modern SOTA model is far more aggressive with its integration tests, often going to a lot of effort to set up sandboxes for integration tests.
I suspect this is a major reason for today's models being far better programmers than earlier ones. But as we all know, tests have a fatal flaw - they can only prove the presence of bugs, not their absence. If you are creating code with a large number of random bugs, relying on tests to catch them all won't work. Instead you start using other tricks such as isolation boundaries, simplifying APIs so they are hard to misuse, strict information hiding - all things an experienced programmer learns over time, because they too write code with a lot of bugs. So far, there is little sign the models use the tricks well. Either they don't know them, or their stochastic nature means they can't implement them reliably.