GPT-3 doesn't know how to code. However, it has so many parameters that it was almost able to memorize its training data, which included people asking how to code something and other people answering.
Even if it was completely rote memorization it is still extremely valuable to be able to give a plain text prompt and get relevant answers back. As SWEs most of us probably have very high end google-fu so we can find potentially obscure answers to our questions but this could make that kind of skill redundant. Why try to search stackoverflow when GPT could just generate exactly the code snippet you need?
I see people saying this over and over again, with nothing to back it up. I could say the same about the human mind.
And at least those can be run on prem. This thing we are talking about is already in walled gardens.