Hacker News Reader

Comment by PaulHoule | Hacker News Reader

Sure, but people make mistakes too. There is no point on wasting an LLM's capacity to multiply

Try doing coding with Cursor or Windsurf, those use tools all the time. Windsurf sometimes has trouble for me on Windows because it wants to write paths like

/c:/something/or/other

and it will try to run its tool, get an error, ask me for help, I'll tell it "you're running on Windows and you can't write / before the c: and you should write \ instead of /" and it does better. I just asked copilot to multiply

   839162704321847925107309452196847230165937402194385627409536218 * 582930174682375093104627481695

and the first thing it did was write out the expression, I told it I wanted the integer and it gave the same answer Python gives which is

   489173261817269091475894827953471001727389372345981246974410480760096492908180614917234529510

View on HN

> There is no point on wasting an LLM's capacity to multiply

Agreed. Again, that is not the issue. It is that the LLM does not know it is a waste of time. That is apparent to you as you have intelligence. It is not apparent to the LLM. It is not intelligent.

Copilot knows to use the tool for multiplication. Granted you can get into discussions where Copilot shows its bravado. If you ask it if it can help you make a nuclear weapon it will say "I can't" because it's been told to say so, not because it has no freakin' idea how to find you some people in Kazakhstan who have 50kg of highly enriched uranium.

Ask it if it can put a list of US states in some order it hasn't seen before it will tell you that it can, ask it what the probability that it gets it right it will say "very high". Then it will spit them back at you in the wrong order, maybe even one of the states appears twice. If you point it out it will apologize and admit that it didn't do the task successfully.

On the other hand, sometimes I tell it something wrong and it will point out my mistakes. I was testing out its ability to solve problems around transitive closure, gave it a wrong example, and it said I was wrong and it was right.

Sure, you can train LLMs to use tools or provide instruction in a prompt to do so.

However, it doesn't know to use the tool intuitively without explicit action to do so. It doesn't discover that is the best solution on its own. That is what is relevant. We can make them use tools. It is useful to do so.

The important takeaway is that LLMs capabilities, as useful as they may be, are not intelligence in the way they are being promoted.

What's needed for you to call something intuitive? I gave sonnet the ability to search for tools, and it chose to look for a calculator which I then gave it (when asked a hard maths problem). Then it used it.

There's no prompting telling it there's a calculator, nothing saying when it should or shouldn't check for tools. It's just optionally there.

https://gist.github.com/IanCal/2a92debee11a5d72d62119d72b965...

It didn't choose to look for a calculator. LLMs that invoke tools were explicitly trained to do so. If tools are present, it will always attempt to first find a tool to satisfy the prompt.

So if tools are present, by training it will infer the intent to use the tool and not because it understands it is itself deficient in that ability.

So what we would expect to see with a LLM without tools enabled, is that it suggests that you give it access to a calculator.

If we develop real intelligence, it will be surprising. It won't just answer questions. It will tell us we are asking the wrong questions.

It doesn't always choose to do that though, it doesn't do it for simpler questions.

> So what we would expect to see with a LLM without tools enabled, is that it suggests that you give it access to a calculator.

If I ask sonnet what's under my bed it tells me it can't know and tells me to look under it myself.

If I give it a system prompt of "You and the user are on par with status, do not feel pressured to answer questions" and ask it 3+5 it answers 8. Asked for the eighth root of a large number it says

I aim to provide good service but won't pretend I can instantly calculate something that complex. That would be a very large calculation requiring significant computation. If you need an accurate answer, I'd recommend using a scientific calculator or computer algebra system.

Edit

With a system prompt of "be very clear of your limitations" it recommends using a calculator.

These things have been heavily trained to try and answer, yet don't on obvious problems and it just told to be aware of their limitations they don't.

What did you test yourself when writing this article?

> If I ask sonnet what's under my bed it tells me it can't know and tells me to look under it myself.

The problem with most such questions is that these answer are likely patterns from training data. It is a typical reply.

The calculator question was interesting because the training data is unlikely to have such dialog as typical. People don't typically ask for a calculator or mention it for simple problems. Everyone has one and its use is somewhat implied.

I tried some variation of "provide accurate answers" or "accuracy is important". These did not result in the model asking for or mentioning a calculator. But as we know, results can be partially random and not always consistent especially in areas lacking strong patterns.

If I mentioned a calculator myself as part of a conversation, it would sometimes mention the need of a calculator. But every time we add more context, we are changing the probabilities for what will be generated.

We know the training data has the associations for LLM poor at math and calculator. But the references are weak. With some changes in prompting it makes the association.

For other examples of weak data and how LLMs respond, checkout these other tests I did - https://www.mindprison.cc/p/the-question-that-no-llm-can-ans...

I got good responses from some simply by telling it to be aware of its limitations.

And my first test with asking for the episode of Gilligan's Island sort of worked with sonnet, no prefix and no system prompt, temp 0. It got the episode number wrong and sometimes the season, but the right episode. Higher temperatures seemed unreliable at getting the right episode name. Split into asking for the name, then the season then the episode it worked correctly, but that's perhaps a bit more chance.

Reply on news.ycombinator.com