If you're interested in this you should take a look at Chrome's break engine source code for japanese/chinese: https://github.com/unicode-org/icu/blob/778d0a6d1d46faa724ea... https://chromium.googlesource.com/v8/v8/+/refs/heads/master/...
https://github.com/polm/fugashi/issues/23
(Also, if you learned about Chrome's Japaense tokenization on HN, that was me too!)