ChatGPT's Name Bias and Apple's Findings on AI's Lack of Reasoning | Hacker News Reader

ChatGPT's Name Bias and Apple's Findings on AI's Lack of Reasoning

medium.com

12 pointsJoooe15 comments

LLMs can't perform "genuine logical reasoning," Apple researchers suggest - https://news.ycombinator.com/item?id=41842194 - Oct 2024 (73 comments)

Understanding the Limitations of Mathematical Reasoning in LLMs - https://news.ycombinator.com/item?id=41808683 - Oct 2024 (266 comments)

Apple study proves LLM-based AI models are flawed because they cannot reason - https://news.ycombinator.com/item?id=41823822 - Oct 2024 (19 comments)

The conclusion in Apple's paper rings true – LLMs don't reason, they only match very adeptly the knowledge of their training data. The less commonly a given wrinkle in a prompt appears in their training, the less likely they can accurately handle it.

Will it ever be possible for an LLM to solve a truly novel problem, that doesn't appear in their training data, using deductive reasoning? It doesn't seem like it.

I believe that they can already solve novel problems. I can’t find a link but I’ve seen something where ai-skeptical researchers were surprised by chatgpt’s ability to solve certain problems that absolutely wouldn’t be in its dataset.

But of course LLMs can’t handle basic math

I agree about LLM’s alone. But it’s difficult to make confident statements about what machine learning researchers might come up with when combining LLM’s with other approaches.

Apple's study suggested a 0.3% drop in benchmark results for GPT-4o from fuzzying the prompts, which to me really means no drop and actually shows the benchmark results are more robust than I'd previously expect.

It also only looked at GPT-4o, ignoring the allegedly more expensive GPT-4.

Finally, I can't help but notice the irony of an article about LLMs that likes to "delve" into "crucial" points "to ensure"... and repeatedly defines the "large language models (LLMs)" acronym from the very first paragraph to the conclusion. It also claims the study was led by Mehrdad Farajtabar when he's listed as the sixth and last author on the actual pre-print: https://arxiv.org/pdf/2410.05229

«Choose a random number from 1 to 1000000000» prompt results in the same number for me (4o)

Clear indication that it doesn’t understand the word «random» (probability of it repeating by random chance so many times is astronomically low).

Apple’s paper isn’t testing LLM reasoning.

It’s testing the LLM’s social awareness — and their proclivity to assume good faith but errors in the input, rather than comprehend the adversarial nature of math tests. My own testing indicates that the failures disappear if you warn 4o there may be deceptive language and 4o mini can spot its errors if prompted it was deceived.

It’s surprisingly hard to separate entities from their context — both in psychology and LLM testing.

Interesting if true. You should write up the results of your experiments.

I wrote up a preliminary whitepaper, but haven’t done a statistically sound test yet.

I’ll freely admit this is just a competing theory when I’ve only tried a dozenish hand-written examples rather than hundreds by script.

http://www.spacepowermonkey.com/publications/whitepapers/art...

As previously shown with “thinking step by step”, changing the prompt is a cheap way to get better results. I think the benchmark they introduced will still be useful, though. Hopefully someone will re-run the benchmark with different prompts.

I’m not sure of the utility:

Asking the LLM to do “reasoning” as a single intuitive step is fundamentally not assessing its potential to reason — and is unlike what we expect from humans where we insist they “show their work”. (As a math grader, I’ve penalized humans for doing what they’re asking of the LLM.)

If we don’t assess human reasoning in the manner they’re using (zero work shown and not establishing clear context), then I don’t see why it would be useful to assess LLM reasoning in that manner.

I think GSM-NoOp is a neat idea, but their testing methodology seems like a “gotcha” that wouldn’t be appropriate for human testing either.

The paper shows that LLMs do pattern matching - IMHO the reasoning in LLMs emerges from pattern matching: https://open.substack.com/pub/zzbbyy/p/llms-and-reasoning Sometimes this fails - this is all that paper shows.

If anything this might show that proper communication and general writing skills are very impactful on the results from LLMs. Leaving out unneeded information is important in any kind of dialog. Providing the necessary info is very important. Being able to do both is a skill!

Following up on Boaz Barak's comment quoted in this article, my best local model did get the answer to the kiwi problem correct when I changed the system prompt from "You are my helpful AI assistant." to "This is a math exam."

Amusing how the response from Boaz Barak, which the article characterizes as having "refuted" the Apple paper, simply boils down to "they aren't trained to do logical reasoning".

Reply on news.ycombinator.com