Hacker News Reader

Comment by rwmj | Hacker News Reader

Can modern humans count the number of tokens in "blueberry"? (You're not allowed to cheat and use https://platform.openai.com/tokenizer)

View on HN

I see this argument every time this particular failing is brought up and like, regardless of the reasoning, it's still completely failing to do something a second grader should be able to do and until it stops failing in that way, being like "Um technically it's a lot harder than you think" isn't going to win over AI skeptics.

A Chinese second grader would probably not be able to answer "How many Rs are in the word 草莓?" It's a valid critique.

You're right they probably wouldn't be able to to it, but that child isn't being marketed as an invaluable business tool to assist doctors in diagnosing cancer, or whatever.

Sure.

A non-English speaking adult wouldn't be able to answer the question either, after the question was translated to their language. Maybe you wouldn't want a non-English speaker helping you to write an acrostic in English. Luckily nobody is marketing LLMs as "great for designing word puzzles" though.

You can make comparisons all you want but you can't change the fact that it just plain looks bad.

One is an unambiguous question, the other depends on the tokenizing algorithm.

No, but I can recognize my inability to do so instead of making the answer up.

Uh, who cares?

That is literally useless knowledge.

When's the last time you needed to know the number of Rs in "blueberry"?

When's the last time you needed to count the number of things inside a larger thing?

What's your point? Parent said "That's literally useless knowledge" in response to not knowing the number of tokens in "blueberry". Are we talking about the specific question or the general notion of counting?

The point is that LLMs don't speak / think in English. Asking them about spelling is like asking a Chinese speaker, during a text chat with translation, about English spelling. We can give the Chinese speaker access to an app to translate back to English so they can answer these questions. But they (the LLM) don't currently have access to that.

> The point is that LLMs don't speak / think in English.

They don't think or speak anything at all. They use a statistical model to predict the next most likely token to display in response to a prompt.

Distinction without a difference.

s/speak \/ think/operate/

If that makes you happy.

There is an enormous difference between claiming an LLM can think or use statistics to generate a token. All of the difference.

Irrelevant to the discussion at hand + everyone knows what you mean when you refer to "how an LLM thinks" + prove the difference between one statistical process generating tokens (LLM) and another statistical process generating tokens (human brain) please.

Reply on news.ycombinator.com