Go look at the GPT training code, here is the exact line: https://github.com/karpathy/nanoGPT/blob/master/train.py#L12...
The model is only trained to predict the next token. The training regime is purely next-token prediction. There is no loopiness whatsoever here, strange or ordinary.
Just because you take that feedforward neural network and wrap it in a loop to feed it its own output does not change the architecture of the neural net itself. The neural network was trained in one direction and runs in one direction. Hofstadter is surprised that such an architecture yields something that looks like intelligence.
He specifically used the correct term "feedforward" to constrast with recurrent neural networks, which GPT is not: https://en.wikipedia.org/wiki/Feedforward_neural_network