From the OP, it sounds like those tokens are generated from the UTF-8
bytes instead of from the Unicode code points. And those bytes are, taken in isolation, complete nonsense. Imagine a token that represented the right side of the letter d followed by the left side of the letter e but could also represent other mishmashes of characters.
I bet the first layer of the model is mostly stuck reconstructing something resembling actual words.
(UTF-8 is locally decidable. I bet that a bit of work on the token list could cause it to avoid tokens that do not align with code point boundaries.)