It feels a bit like someone saying ‘this isn’t a fundamental limitation of cameras; the lens is the problem’… but I’m very much ignorant about these things so I’d be happy to have it explained.
It feels a bit like someone saying ‘this isn’t a fundamental limitation of cameras; the lens is the problem’… but I’m very much ignorant about these things so I’d be happy to have it explained.
> In this case it would be pretty easy for the LLM to reason about how often t appeared in the sentence.
I’m not sure that follows though; is it true that LLMs already find it easy to ‘reason’ about how often (token) appears in some given text? I don’t think so.
Also, tokens are just numbers internally after tokenization. So LLM sees the question as "how many letters N are there in 563 3684 11267?". It has basically no idea what you're talking about. Unless somewhere in the training data someone mentions that token number 11267 has two letters N.
It’s not just the latter part of that sentence that consists of ‘numbers’, all of it does.