Hacker News Reader

Comment by csallen | Hacker News Reader

> remember when AI couldn’t count the number of Rs in “strawberry”?

GPT-5 still gets this wrong occasionally. Source: I just asked it: How many r's are in "strawberry"?

It said 2.

(I dislike this method of testing LLMs, as it exploits a very specific and quirky limitation they have, rather than assessing their general usefulness. But still, I couldn't resist.)

View on HN

My favorite test is to ask it to invent a magic trick given a set of constraints and props. Because magic is generally published very secretly, surprisingly little of it is in most training sets. Pretty much just the most common method & gimmick exposures people tend to parrot online, but not the theory or exact routines behind those methods.

The worse an LLM is, the more likely it is to suggest literally impossible actions in the method, like “turn the card over twice to show that it now has three sides. Your spectators can examine the three-sided card.” It can’t tell logic from fantasy, or method from effect.

I would love to see a trick where you keep turning the same card over with different faces. Bill it as a card trick with just one card. I can think of a half dozen good ways to do it (and you'd need more than one).

That's kind of magician thinking though. I'm not sure it's any better than the Elmsley's Dazzle.

There is a very basic three card trick. You turn each card up one at a time, showing that all three cards are the same(say king spades). End with all three face down in your hand. Then you show all three cards again and all three are a different card(king hearts). At no time are there more or less than 3 cards in your hand.

I'm a magician. There is a magician AI that is seeded with a ton of tricks that can perform this activity.

But it's all context after the fact. There's very little an LLM is going to have about that context as you rightly pointed out.

I've come to realize that asking an LLM to do something that a purpose-built software program can already do is a mistake. Instead, the LLM must be an agent for that computer program.

Therefore, the correct prompt is "write a python program to count the number of letters in a word, and then use it to count the number of Rs in strawberry".

Remember those interview questions: "how many golf balls fit in a 747?" Turns out they are actually a good way to see if the candidate is a LLM-style imitator or is a someone who knows how to switch to a formal constraints-based thinking, come up with that "python program" to count golf balls and execute it step-by-step.

I wouldn't consider that a measure of it's usefulness or lack thereof, but it is an indicator that LLMs continue to not actually work like a human brain, which isn't surprising if you know the technology but might be to the lay user.

“There are 2 — the r’s are in strawberry (letters 3 and 8–9 form the double r).“

That took 4 seconds

What a waste of resources

Is this really a quirky limitation? LLMs seem to struggle with any sort of task requiring a "complete" list of anything. Which is a massive problem because those are super common tasks.

if the router is working properly, it should send questions like this to thinking mode which can handle it well. unfortunately the router doesn't always seem to make the right call.

> how many r's are there in "strawberry"?

>> 3

This was for GPT-5 regular

Reply on news.ycombinator.com