In your comment above:
(...) is expressed a little bit more clearly as _the LLM is learning an n-gram_ that produces correct sorts (...)
(My underlining)
You also use it in a similarly unusual way throughout your linked substack post, for example, you write:
the way GPT works is, in a certain sense, functionally equivalent to an n-gram, but that doesn’t mean GPT is an n-gram.
Where does this use of "n-gram" come from? I mean, did you see it somewhere? I'm curious, where?
>> The tokenizer would throw an exception, because it doesn't have any tokens to represent alphabetical characters. But you tell me - if I had tokenized alphabetical characters and defined an ordering, would you expect the results to be any different?
I'm sorry, I don't understand. "Defined an ordering", where?
You can change your tokenizer but that will not change the trained model, obviously. So if you take your model that's trained on two-digit lists of integers and you run it on lists of any other type of elements it will not be able to sort them correctly. But isn't that what you claim? That:
"the LLM's training molds it into representing an actual sorting algorithm that would correctly generalize to any input list"
"Any input list"? How so?