> But nobody knows how GPT works. They know how it was trained, because the training scheme was designed by humans, but the algorithm that is executed during inference was not intelligently designed but evolved, and it is implicit in the structure of the network, and interpretability has yet to mature to the point where we can draw a symbolic, abstract, human-readable program out of a sea of weights.

Nobody knows how the human mind really works either. And we’ve been trying to understand ourselves for thousands of years. I suspect we will take a while to figure out how the “mind” of GPT works too.