So perhaps human language is simply never complex enough to need more than 3072 numbers to represent a given train of thought, but that doesn't seem clear to me.
Will compute allow that number to go up? Or is that an optimal number?
Will compute allow that number to go up? Or is that an optimal number?
In general for models we know its trended upward, but for sure it’d be interesting to know what they’re using now.