Hacker News Reader

Comment by planb | Hacker News Reader

Please anyone correct me if I'm wrong: LLMs cannot solve this kind of riddle. This has nothing to do with their capabilities for logical reasoning, but with the way words are represented as tokens. While they might know that "apples" has two syllables because that is mentioned somewhere in their training data, if you make up a fruit "bratush" a human will see that as two syllables, but this might be 1 to 7 tokens to a LLM without any information about the word itself.

View on HN

Well I tried it out in GPT4 with made up words-

Tabitha likes bratush but not zot. She likes protel but not kig, and she likes motsic but not pez. Following the same rule, will she like tridos or kip

Given the examples, one speculative pattern could be that Tabitha likes words with at least two syllables or a certain complexity in structure. Therefore, following this speculative rule, Tabitha might like “tridos” more than “kip.”

Zot is a word already being used in the world: verb. (slang) To zap, kill, or destroy.

So is protel: https://en.m.wikipedia.org/wiki/Protel

So is kig: https://en.m.wiktionary.org/wiki/kig

Pez is a well known brand name in America.

Kip is commonly used name in America.

Motsic is a fairly common last name from searching.

Tridos is used all over the internet as a brand name so this all seems probable to be in the training data.

These words are not new nor are they made up.

LLM can solve this for all tokens where it got to learn how many syllables are in that token or a combination. If you trained it to work on single letters only it would do better at that task than word chunks (same for math and single digits). It will generalise to new words if the token level knowledge is there.

Whether this means it can or cannot solve that kind of riddle is up for your interpretation. I understand square root and can calculate square root of 16, but not of 738284.7280594873. (in a reasonable, bounded time) Can I solve square roots?

GPT-4 has a really good tokenizer that is able to retain and use more information about input tokens than one might naively think.

Interesting. Given the example with made up words someone posted here it really looks like I was wrong.

The amazing thing about emergent behavior in LLMs is that they are able to answer questions like these. I don't think it is completely understood how exactly they do this, but there's little doubt that they do.

Do you have any sources that prove this is true?

This looks pretty good to me:

https://chat.openai.com/share/040ac123-c690-4274-8216-6ae091...

Your word is not made up, nor are some of the others in the sample like”pez.” I don’t think this test proves what you think it does. Bratush come up quite a bit in the internet: https://www.thefreedictionary.com/words-that-start-with-brat...

ChatGPT knows how to answer this question, but how I don't know. Perhaps it's programmed to answer that question?

Reply on news.ycombinator.com