I realize that these models are more than powerful enough to deal with this nonsense, but it seems like, especially for smaller models, it might make sense to try using the Unicode input as such instead of treating it as bytes.
I realize that these models are more than powerful enough to deal with this nonsense, but it seems like, especially for smaller models, it might make sense to try using the Unicode input as such instead of treating it as bytes.
There's an idea that you can generalize concepts among different languages, and that you'll benefit from the extended training corpus. As in, talking about an idea from different perspectives helps the model carve it out. But I don't have anything concrete to back that claim up.
[0] but it might be worth it if you need a smaller model because then there are tradeoffs again.
I bet the first layer of the model is mostly stuck reconstructing something resembling actual words.
(UTF-8 is locally decidable. I bet that a bit of work on the token list could cause it to avoid tokens that do not align with code point boundaries.)
There is over 1M possible Unicode code points, and 150k actually defined. Thus, you can't really encode all of them with splitting.
Then you could have the "how many r in strawberry" equivalent of "how many 月 in 明月清风"! On the negative side, a model trained on such a representation could make up CJK characters not in Unicode and you would need a procedural font to display them properly.
This is one thing among many done by our llguidance [0] library.
[0] https://github.com/microsoft/llguidance
edit: if anyone's interested:
(([C2-DF] [80-BF]) | (E0 [A0-BF] [80-BF]) | ([E1-EC] [80-BF] [80-BF]) | (ED [80-9F] [80-BF]) | ([EE-EF] [80-BF] [80-BF]) | (F0 [90-BF] [80-BF] [80-BF]) | ([F1-F3] [80-BF] [80-BF] [80-BF]) | (F4 [80-8F] [80-BF] [80-BF]) | [00-7F])
When you tokenize “ant” to 0x38 0xF9, it doesn’t matter if the original was three bytes of ascii or 0x00 0x00 0x00 0x61 0x00 0x00 0x00 0x6E 0x00 0x00 0x00 0x74
Models will generally only produce valid UTF8 (that is when bytes of tokens are concatenated they are valid UTF8), unless really confused.
What about English? Just as there is no natural boundary between tokens in English, there is no natural boundary between words in Chinese. Before LLM became popular, people had invented many ways to do Chinese word segmentation, just like nowadays people are inventing many ways to do tokenization.
However in the past, most of the time, you would end up with ngrams. If we learn that from history, ngrams should be a good starting point for English. For example, word "token" should be 3 tokens, "tok", "oke", "ken". Once add Chinese, everything should be just fine.
To be more controversial, I would say there is no such a language called Chinese. They are a group of languages who adopted Universal Token. Now it is time for English to jump on the bandwagon.