For the record, you can do the same things with a Hidden Markov Model (or hand-crafted rules) and the results won't be very different. Except that they won't elicit breatheless articles about being a "step towards general intelligence".
Not to mention that the text generated by GPT-2 can often fool an online reader whereas HMMs have the problem of being long-term incoherent and don’t reference back to subjects of the sentence like GPT-2 often does.
I’m not staying you should believe the AI hype in news media. But the paper does contain a lot of thorough analysis and comparison to the previous state of the art.
https://cdn.openai.com/better-language-models/language_model...
I guess people think "it's a powerful model so it should do well in any task" but that's typically not the case for neural nets. I know what OpenAI claims about how it can do a little bit of everything, machine translation benchmarks are borked and I bet so are question answering ones (which I confess I don't know much about).