[0] but it might be worth it if you need a smaller model because then there are tradeoffs again.
There's an idea that you can generalize concepts among different languages, and that you'll benefit from the extended training corpus. As in, talking about an idea from different perspectives helps the model carve it out. But I don't have anything concrete to back that claim up.
I bet the first layer of the model is mostly stuck reconstructing something resembling actual words.
(UTF-8 is locally decidable. I bet that a bit of work on the token list could cause it to avoid tokens that do not align with code point boundaries.)
This is one thing among many done by our llguidance [0] library.
[0] https://github.com/microsoft/llguidance
edit: if anyone's interested:
(([C2-DF] [80-BF]) | (E0 [A0-BF] [80-BF]) | ([E1-EC] [80-BF] [80-BF]) | (ED [80-9F] [80-BF]) | ([EE-EF] [80-BF] [80-BF]) | (F0 [90-BF] [80-BF] [80-BF]) | ([F1-F3] [80-BF] [80-BF] [80-BF]) | (F4 [80-8F] [80-BF] [80-BF]) | [00-7F])
There is over 1M possible Unicode code points, and 150k actually defined. Thus, you can't really encode all of them with splitting.
Then you could have the "how many r in strawberry" equivalent of "how many 月 in 明月清风"! On the negative side, a model trained on such a representation could make up CJK characters not in Unicode and you would need a procedural font to display them properly.