(my opinion) It is not predicting based on 'words/tokens'. It is transforming the general words/tokens embeddings into a context specific embedding which encodes "meaning". It is not an n-gram model of words. It is more like an n-gram model of "meaning". It doesn't encode all the "meanings" that humans are able to but with addition labelled data it should get closer. I think gpt is a component which can be combined to create AGI. Adding the API so it can use tools and allowing it to self-reflect seem like it will get closer to AGI quickly. I think allowing to read/write state will make it conscious. Creating the additional labels it needs will take time but it can do that on its own (similar to alpha-go self-play).