Are you saying that "a lot" isn't almost semi-trivial to obtain, seeing how much code is available online?
And while it makes tons of obvious mistakes in English (which is a much more flexible and forgiving language), its PHP is somehow nearly syntactically perfect?
-
Examples from GPT-2 GitHub have a lot of code:
https://raw.githubusercontent.com/openai/gpt-2/master/gpt-2-...
To me, this doesn't seem like an argument in favor of this model "understanding" English (or C, or PHP). It seems more like an indication that it memorizes way more information than the paper implies and then does clever word substitution.
See also the example in the paper of accidentally learning to translate into French even though they tried to remove French pages from the corpus.
I'm not sure what point you're trying to make. Do you think a neural net is not capable of generating the code in the gist? Because it's pretty easy to do that. The harder part that we're still trying to figure out is getting that code to do something meaningful.