I have two tasks I wanted it to try, both making use of public APIs, starting from scratch. In short, it was frustrating as hell. Never-ending import problems -- I'd tell it the error, it'd give me a different way to import, only leading to a new import problem. I think I used up all my 100 queries in 4 hours of GPT-4 just on the import/library problem.
Then there were constant mis-use of functions -- ones that didn't exist, or didn't exist in the object it was using, but did exist in some other object instead, at which point it would apologize and fix it (why didn't you give it to me correct the first time, if you "know" the right one?)
The actual code it wrote seemed fine, but not what I'd call "scary impressive." It also kept writing the same code in many different styles, which is kind of neat, but I found one style I particularly liked and I don't know how to tell it to use that style.
Lastly, it's only trained up to Sep 2021, so all the APIs it knew were well behind. I did manage to tell it to use an updated version, and it seemed to oblige, but I don't really know if it's using it or not -- I still continued to have all the above problems with it using the updated API version.
Anyway, I hope MS fiddles with it and incorporates it into Visual Studio Code in some clever way. For now, I'll continue to play with it, but I don't expect great things.