That is exactly what I mean. GPT infers patterns. Some language patterns themselves are categorization, like words surrounding "or". It's important to recognize that humans are creating the categorization in these cases.
GPT does not differentiate at all between one pattern and another. It doesn't categorize the pattern that represents words, or the pattern that represents sentences, or the one that represents an answer in response to a question, etc. These are all symbolic categorizations that humans do to text; and we call that set of categories "language". GPT, ironically, has no concept of language or grammar. It only sees patterns as patterns.