Too many assumptions.
>> Do you have some competing idea of how it works? It would behoove you to share it with the thread.
Yes, I do. It's a language model. It is trained to model the text in a corpus. There's no mystery there and there's no reason to assume it's doing anything else than what it was designed to do. It's a machine. People made it. It does what it was made to do. Other explanations of its behaviour are fantasies.
And computers aren't brains. Brains may be computers, or they may not, but you can't look at brains and draw conclusions about computers. For the same reason that you can't look at planes and draw conclusions about birds.
>> The use of backpropagation and gradient descent allows us to search the space of all possible models starting from a randomized model, and find a model that does this by nudging our model to be a little more capable at it over and over again.
If you haven't already, you should try implementing backpropagation, or even just gradient descent/hill climb, and observe its behaviour for a while. I suspect that, if you haven't done that, the experience will clear a lot of the rosy clouds about what backpropagation and gradient optimisation can really do.
Anyway this is getting a little too long-winded for me and I don't have time to continue the conversation. I must respectfully bow out. Last word is yours and thanks for the interaction.