The bigger issue for me has been hallucinated libraries. It will link to things that don't exist that it insists do. Sometimes I've been able to get it to output the library it hallucinated though.
It also makes more syntax errors than a person in my experience, but that is made up for by it being really good at troubleshooting bugs and the speed it outputs.
A good example looks like:
``` I am trying to model associative memory that i may attach to a gpt model
here is the code for the memory:
....
can we keep the input vectors in another array so we can fetch items from it directly instead of having to reconstruct? ```
I had a spooky experience with a project that was written almost entirely by GPT. I gave it the skeleton and one method and asked it for a modification. It gave it and also said "don't forget to update this other method", and showed me the updated code for that too.
The spooky part is, I never told it the code for that method, but it was able to tell from context what it should be. (I told it that it itself had written it, but I don't know if that made any difference: does GPT know how it "would" have done things, i.e. can predict any code it knows that it wrote?)
Often, it will posit the existence of a function that is named slightly differently. This is great for helping you find corners of an API with functionality you didn't know existed, but insanely frustrating when you're just trying to get code working the first time. You end up manually verifying the API calls.