GPT-3 is taking a graph-structured object ("language" inclusive of syntax and semantics) over a variable-length discrete domain and crushing it into a high-dimensional vector in a continuous euclidean space. That's like fitting the 3-d spherical earth onto a 2-d map; any way you do it you do violence to the map.
I think systems like GPT-3 are approaching an asymptote. You could put 10x the resources in and get 10% better results, another 10x and get 1% better results, something like that.
You might do better with multi-task learning oriented towards specific useful functions (e.g. "is this period the end of a sentence?") but the training problem for GPT-3 is by no means sufficient for text understanding.
GPT-3 fascinates people for various reasons, one of them being almost good enough at language, lacking understanding, faking it, and being the butt of a joke.
If GPT-3 were a person with similar language skills and people blogged about that person, mocking it's output, the way we do with GPT-3, people would find that cringeworthy. Neurotypicals welcome it as one of their own, and aspies envy it because it can pass better than they can.
At $2 a page it can replace richmansplainers such as Graham and Thiel who never listen. It's not a solution for folks like like Phillip Greenspun who read the comments on their blogs.
For that matter, it may very well model the mindlessness of corporate America: if you accept GPT-3 you prove you will see the Emperor's clothes no matter how buck naked he is. AT&T executives had a perfectly good mobile phone business: what possessed them to buy a failing satellite TV business? Could GPT-3 replace that "thinking" at $2 a page? Such a bargain.