If every pair of digits appears sorted in the dataset, then that could still be “just” a stochastic parrot.
I’m kind of interested to see if an LLM can sort when the dataset specifically omits comparisons between certain pairs of numbers.
Also I don’t think OC was responding to commenters, but the article
But by specifically avoiding certain cases, wet could verify if the model is generalizing or not.
As for avoiding certain cases, that could be done to some extent. But remember that the untrained transformer has no preconception of numbers or ordering (it doesn't use the hardware ALU or integer data type) so there has to be enough data in the training set to learn 0<1<2<3<4<5<6, etc.
This is the kind of thing I’d want it to generalize.
If I avoid having 2 and 6 in the same unsorted list in the training set, will sets containing those numbers be correctly sorted in the same list in the test set and at the same rate as other lists.
My intuition is that, yes, it would. But it’d be nice to see and would be a clear demonstration of the ability to generalize at all.