Comment by PaulHoule | Hacker News Reader

Comment by PaulHoule | Hacker News Reader

Parent Full thread

That would require the models to have real reasoning ability as opposed to looking up preformed answers on Stack Overflow.

Study after study shows that many evaluations of LLMs are deceptive because they evaluate them on questions that were answered in the training set. On the other hand, this is exactly why they can do so well on medical board exams because you don't learn to pass the medical board by predicting what medical treatments should work according to first principles but instead by remembering which medical treatments have been proven to work.

For that matter I've long wanted a conventional search engine which can be specialized for a particular software project I am working on: for instance I might be using JDK 17 and JooQ version 3.15 and I only want to search those versions of docs. It shouldn't be hard to look at the POM file to figure these versions out. I have the exact same problem with Javascript where I have to work with code that uses different versions of react-router, bootstrap, MUI, etc. The idea LLM coding assistant should do the same.

I just asked GPT 4o to do the following and it succeeded:

> Write me JS code that generates a random list of 10 prime numbers, takes the sine of each (as radians), then takes the cosine of each of those values (as degrees) and then adds 5 to each value

I assure you, this task is not on Stackoverflow (because nobody would ever be deranged enough to want to do this) but GPT 4o could do it!

Admittedly, this shows only a very rudimentary reasoning ability but it demonstrates some. The idea that LLMs are just a slight evolution of search indexes is demonstrably false.

This takes 0 reasoning. You’ve given it an exact map to follow to the answer.

LLMs are also good at tasks that are roughly "linear" in the sense that a group of input tokens corresponds to a group of output tokens and that translation moves from left to right.

In a hard programming problem, for instance, you have a number of actions that have to take place in a certain dependency order that you could resolve by topological sort, but in a problem like the above one bit of English corresponds to one bit of Python in basically the same order. Similarly if it couldn't translate

    take the sine of...

to

    Math.sin(x)

because the language didn't already have a sin function it would have to code one up. Similarly, translating between Chinese and English isn't really that hard because it is mostly a linear problem, people will accept some errors, and it's a bit of an ill-defined problem anyway. (Who's going to be mad if one word out of 50 is wrong whereas getting one word wrong in a program could mean it doesn't compile and delivers zero value?)

LLMs can do a bit of generalization beyond full text search, but people really underestimate how much they fake reasoning by using memory and generalization and how asking them problems that aren't structurally close to problems in the training set reveals their weakness. Studies show that LLMs aren't robust at all to the changes in the order of parts of problems, for instance.

The biggest problem people have reasoning about ChatGPT is that they seem to have a strong psychological need to credit it with more intelligence than it has the same way we're inclined to see a face inside a cut stem. It can do an awful lot, but it cheats pervasively and if you don't let it cheat it doesn't seem as smart anymore.

Reply on news.ycombinator.com