imho the fact that character level language models work mostly just as well as word level or bpe tokenizers makes it clear that the "embedding" view is wrong nowadays, that the meaning is really built (and stored) in the layers, just fine
How are tokens/phrases/documents represented, computationaly? Theoretically, do they live in a vector space?