Hacker News Reader

The problem is that detecting the different variants to label the training data requires understanding the semantics (detecting French vs English can be done purely lexically), which has historically been a hard problem. With modern language modeling it’s tractable.

View on HN

I think Google has the resources required to detect different languages in Chinese script (different combinations of characters and vocabulary, even specific Unicode characters, etc.) but there is 1) lack of incentives and 2) opposing political interests. Doesn’t help that there is a bit of a mess when it comes to Chinese characters, like a single Unicode point can be rendered differently depending on the font.

Reply on news.ycombinator.com