For something to generate text that is believable on a level like gpt-4 does it needs to have some model of the real world and understand relationships between things. So yes the training goal was to "just predict the next word" same as our training goal was just "reproduce and survive". What emerged out of that is the real important thing.

No human can understand what really happens in the billion of calculations done for each token in gpt-4 so how can you claim that there is surely no thought process going on? It can solve some riddles, it can draw pictures and it can reason (to some extend). How is that just generating text to you?

In the end this argument doesn't matter because how it was made is irrelevant. What matters is what it is and can do.