I didn't say anything about expertise, I do not use LLM for work and have a background in statistics/teaching. Also, none of this is related to tokenization. It's just the way the application layer infers response.

Any inference or probability engine, be it an app or a heuristic mental process, is obviously based on categorizing historical data and literally weighing it's confidence around it to form responses.

Confidence intervals and associated indexes/logic are the foundation for most heuristic data models.