Hacker News Reader

I'd love to see some examples of frontier models getting letter counting wrong. Can you share some?

I just got Claude Opus to fail at counting letters in a string.

Me: What about in rrorooroororororrrrrrorrororororororrrorooorrorroror

Opus 5.5: There are 29 r's in that string.

There are 30rs in the string. If you want I have a screenshot.

The reality is this, the models "think" in tokens, not in letters, so they can easily make mistakes like this and it's very hard to stop these kind of dumb mistakes happening. So yes, frontier intelligence at work.

Amusingly enough, I corrected it, and it doubled down on there being 29 rs until I forcefully told it that it was wrong... :)

Opus 5.5 medium effort tells me there are 30 rs in that string. Did you change the effort level or something? Also Opus 5.5 isn't a frontier model. Anthropic's frontier model is Mythos/Fable.

You realize that the models are not deterministic right? It can literal vary from attempt to attempt.

Also, Opus 5.5 is considered Anthropics best model right now and if you try it, is better than Fable in almost every task.

You're also splittin hairs if you argue Opus 5.5 isnt "frontier".

https://artificialanalysis.ai/models

Artificial Analysis considers Opus 5.5 to be the best current model, yes better than Fable.

Reply on news.ycombinator.com