> After going in circles a few more times, I decided that was it. It got close. It seemed to understand the problem, but it could not actually properly solve it.

I had this same loop issue with Chat-GPT. I had something I wanted to do with asyncio in Python. That's not something I work with much so I thought I'd see if Chat-GPT could help me out. It was actually good at getting me up to speed on ansycio and which parts of the library to look at to solve my problem. It got pretty close, but it can't seem to solve edge cases at all. I got into this loop where I asked it to make a change and the code it output contained an error. I asked it to fix the error so it gave me a slightly modified version of the code prior to the change. So I asked it to make the change again and the code it spit out gave the same error again. I went through this loop a few times before I gave up.

Overall, it's cool to see the progress, but from what I can tell GPT-4 suffers from all the same issues Chat-GPT did. I think we're probably missing some fundamental advance and just continuing to scale the models isn't going to get us where we want to go.

My biggest concern with the current batch of LLMs is that we're in for Stackoverflow driven development on steroids. There's going to be a ton of code out there copy and pasted from LLMs with subtle or not so subtle bugs that we're going to have to spend a ton of time fixing.