Hacker News Reader

Comment by mg | Hacker News Reader

The prompt I always try first:

    Two cars have a 100 mile race. Car A drives 10
    miles per hour. Car B drives 5 miles per hour,
    but gets a 10 hour headstart. Who wins?

Tried it on Qwen1.5-110B three times. It got it wrong 2 times and correct 1 time.

View on HN

On the contrary I don't really understand what do math problems supposed to prove at this point. We already know LLMs badly suck at math, even if it somehow gets one correct just changing numbers is usually enough to confuse it again. Barring an architectural breakthrough, this next token prediction based AI is very unlikely to get better at doing math.

(Just tested Opus and GPT4-turbo to be sure, both failed. However Llama-3 did get this right, until I scaled up the numbers and then it failed terribly)

I'm mostly a spectator in this, but as someone who hears day in and out about the "AI revolution" I'll point out that the number of places these things are being shoehorned into would greatly benefit from logical consistency, if not the ability to do simple math

Your phrase "next token prediction" is the whole of my heartburn with these stochastic parrots: they can pretend to be good talkers, but cannot walk the walk. It's like conducting interviews or code reviews all day every day when interacting with them: try and spot the lie. Exhausting

Reply on news.ycombinator.com