Hacker News Reader

Comment by tgv | Hacker News Reader

Modern language processing relies on pretty large corpora, and to bootstrap it requires a part to be of high quality and annotated, although for spelling correction you don't need annotation. Or you can go the GPT-3 route and use really huge corpora. If that doesn't exist for your language, and for many smaller languages it doesn't, it won't get better than 2007.

View on HN

Indonesian is spoken by 200M people (45M of which as first language), I bet they could find some text for it. When joined with the Malaysian family, it brings the total speakers to 290M.

My understanding is that the languages with better support have the advantage of official bodies that have section-by-section translations of all their documents; the EU into its 24 official languages and the UN with its six, for starters.

Building corpora has been an ongoing research activity since the 1990s. Many countries lacked the academic programs and support to build them. Often, the material isn't even there. E.g., wikipedia is a great source, but many "national" wikis are rather empty. The Bahasa wiki has 1/10th of the articles of the English one. They also seem to be shorter.

But also, Apple doesn't do much research. MS, however, has always had an active NLP research and support program, if only because of the importance of Word. But Apple depends on the research available from third parties.

I'm not sure about that. It doesn't require a huge corpus to learn the structural features of a language. If you have that plus an English dictionary for the language, an English language model should transfer pretty well to any language.

Reply on news.ycombinator.com