Indeed. I feel like people are not even talking about the same thing in here. I've been using GPT-4 daily for a couple months and it's not outputting code that "only looks correct". It is writing functional solutions that often work as is.

Just yesterday I pasted in a hastily written function and told GPT-4 to "please evaluate this function and make sure it's bulletproof". It outputted a bulleted list of criticisms (mostly true) and then spit out a version of the function that was shorter, easier to understand, and was safer.

That's not just the "next most likely token". Or if it is, then that's all any of us are doing.