Hacker News Reader

It's crazy that OpenAI hasn't fixed their tokenizer yet. They are leaving the door wide open for some Chinese big tech company to capture the non-Latin script parts of the world.

i18n (and accessibility) was something American tech companies were serious about in the 90s and early 2000s. That is how they captured most of the global market. US tech dropping the ball on this leaves the door wide open for Chinese competitors.

View on HN

Does OpenAI’s tokenizer issues cash out into having worse results for Greek rather than just being more expensive for gpt-4? (gpt-3.5-turbo already costs peanuts)

If not, then this response seems overblown. The competitive advantage in LLM at this point probably is not tokenizer optimizations and more about having results worth a damn.

The usability is worse. Token limits are so much easier to reach. It's like using a model with dementia.

I didn't think about that but now it's obvious. Good point.

It could be that they are actively working on this problem, but the product has not yet been released.

Reply on news.ycombinator.com