https://beta.gitsense.com/?chats=ba5f73ac-ad76-45c0-8237-57a...

The left window contains all the models that were asked and the right window contains a summary of the LLM responses. GPT-4o mini got it right but the super majority got it wrong, which is scary.

It wasn't until the LLM was asked to count out the R's that it acknowledges that GPT-4o mini was the only one that got it right.

Edit: I've disabled chatting in the app, since I don't want to rack up a bill. Should have mentioned that.

At a technical level, they don't know because LLMs "think" (I'd really call it something more like "quickly associate" for any pre-o1 model and maybe beyond) in tokens, not letters, so unless their training data contains a representation of each token split into its constituent letters, they are literally incapable of "looking at a word". (I wouldn't be surprised if they'd fare better looking at a screenshot of the word!)

Let me count carefully: s-t-[r]-a-w-b-e-[r]-[r]-y

There are 3 Rs in "strawberry".