Hacker News Reader

Comment by godelski | Hacker News Reader

  > but LLMs do solve problems that people thought were extremely difficult to solve ten years ago.

Well for something to be G or I you need them to solve novel problems. These things have interested most of the Internet and I've yet to see a "reasoning" disentangle memorization from reasoning. Memorization doesn't mean they aren't useful (not sure why this was ever conflated since... Computers are useful...), but it's very different from G or I. And remember that these tools are trained for human preferential output. If humans prefer things to look like reasoning then that's what they optimize. [0]

Sure, maybe your cousin Throckmorton is dumb but that's besides the point.

That said, I see no reason human level cognition is impossible. We're not magic. We're machines that follow the laws of physics. ML systems may be far from capturing what goes on in these computers, but that doesn't mean magic exists.

[0] If it walks like a duck, quacks like a duck, and swims like a duck, and looks like a duck it's probably a duck. But probably doesn't mean it isn't a well made animatronic. We have those too and they'll convince many humans they are ducks. But that doesn't change what's inside. The subtly matters.

View on HN

We don't really have the proper vocabulary to talk about this. Well, we do, but C.S. Peirce's writings are still fairly unknown. In short, there are two fundamentally distinct forms of reasoning.

One is corollarial reasoning. This is the kind of reasoning that follows deductions that directly follow from the premises. This of course includes subsequent deductions that can be made from those deductions. Obviously computers are very good at this sort of thing.

The other is theorematic reasoning. It deals with complexity and creativity. It involves introducing new hypotheses that are not present in the original premises or their corollaries. Computers are not so very good at this sort of thing.

When people say AGI, what they are really talking about is an AI that is capable of theorematic reasoning. The most romanticized example of that of course being the AI that is capable of designing (not aiding humans in designing, that's corollarial!) new more capable AIs.

All of the above is old hat to the AI winter era guys. But amusingly their reputations have been destroyed much the same as Peirce's was, by dissatisfied government bureaucrats.

On the other hand, we did get SQL, which is a direct lineal descendent (as in teacher to teacher) from Peirce's work, so there's that.

We don't have proper language, but certainly we've improved. Even since Peirce. You're right that many people are not well versed in the philosophical and logician discussions as to what reasoning is (and sadly this lack of literature review isn't always common in the ML community), but I'm not convinced Peirce solved it. I do like that there are many different categories of reasoning and subcategories.

  > All of the above is old hat to the AI winter era guys. But amusingly their reputations have been destroyed much the same as Peirce's was, by dissatisfied government bureaucrats.

Yeah, this has been odd. Since a lot of their work has shown to be fruitful once scaled. I do think you need a combination of theory people + those more engineering oriented, but having too much of one is not a good thing. It seems like now we're overcorrecting and the community is trying to kick out the theorists. By saying things like "It's just linear algebra"[0] or "you don't need math"[1] or "they're black boxes". These are unfortunate because they encourage one to not look inside and try to remove the opaqueness. Or to dismiss those that do work on this and are bettering our understanding (sometimes even post hoc saying it was obvious).

It is quite the confusing time. But I'd like to stop all the bullshit and try to actually make AGI. That does require a competition of ideas and not everyone just boarding the hype train or have no careers....

[0] You can assume anyone that says this doesn't know linear algebra

[1] You don't need math to produce good models, but it sure does help you know why your models are wrong (and understanding the meta should make one understand my reference. If you don't, I'm not sure you're qualified for ML research. But that's not a definitive statement either).

> We don't have proper language, but certainly we've improved. Even since Peirce. You're right that many people are not well versed in the philosophical and logician discussions as to what reasoning is (and sadly this lack of literature review isn't always common in the ML community), but I'm not convinced Peirce solved it. I do like that there are many different categories of reasoning and subcategories.

I'd love to hear more about this please, if you're inclined to share.

I'm no expert, but I've been looking into the prospects and mechanisms of automated reasoning using LLMs recently and there's been a lot of work along those lines in the research literature that is pretty interesting, if not enlightening. It seems clear to me that LLMs are not yet capable of understanding simple implication much less full-blown causality. It's also not clear how limited LLMs' cognitive gains will be with so incomplete an understanding as they have of mechanisms behind the world's multitude of intents/goals, actions, and responses. The concepts of cause and effect are learned by every animal (to some degree) and long before language in humans. It forms the basis for all rational thought. Without understanding it natively, what is rationality? I foresee longstanding difficulties for LLMs evolving into truly rational beings until that comprehension is fully realized. And I see no sign of that happening, despite the promises made for o1 and other RL-based reasoners.

Yeah one of the tricky things about causality is that it's not unique. If you didn't record the history of the event then you only have a probabilistic notion of it since many different things and in different permutations could lead to the result you observed. This has led to people believing in multiple universes when it's not akin to there being multiple ways to sum numbers to ten.

I have seen far, far too many people say things along the lines of "Sure, LLMs currently don't seem to be good at [thing LLMs are, at least as of now, fundamentally incapable of], but hey, some people are pretty bad at that sometimes too!"

It demonstrates such a complete misunderstanding of the basic nature of the problem that I am left baffled that some of these people claim to actually be in the machine-learning field themselves.

How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?

I do not know if AGI is possible. Honestly, I'd love to believe that it is. However, it has not remotely been demonstrated that it is possible, and as such, it follows that it cannot have been demonstrated that it is inevitable. If you want to believe that it is inevitable, then I have no quarrel with you; if you want to preach that it is inevitable, and draw specious inferences to "prove" it, then I have a big quarrel with you.

I think the fact that this particular fuzzy statistical analysis tool takes human language as input, and outputs more human language, is really dazzling some folks I’d not have expected to be dazzled by it.

That is quickly becoming the most surprising part of this entire development, to me.

At the very least, the last few years have laid bare some of the notions of what it takes, technically, to reconstruct certain chains of dialog, and how those chains are regarded completely differently as evidence for or against any and all intelligence it does or may take to conjure them.

I'm astounded by them, still! But what is more astounding to me is all the reactions (even many in the "don't reason" camp, which I am part of).

I'm an ML researcher and everyone was shocked when GPT3 came out. It is still impressive, and anyone saying it isn't is not being honest (likely to themselves). But it is amazing to me that "we compressed the entire internet and built a human language interface to access that information" is anything short of mindbogglingly impressive (and RAGs demonstrate how to decrease the lossyness of this compression). It would be complete Sci-Fi not even 10 years ago. I thought it was bad that we make them out to me much more than they are because when you bootstrap like that, you have to make that thing, and fast (e.g. iPhone). But "reasoning" is too big of a promise and we're too far from success. So I'm concerned as a researcher myself, because I like living in the summer. Because I want to work towards AGI. But if a promise is too big and the public realizes it, usually you don't just end up where you were. So it is the duty of any scientist and researcher to prevent their fields from being captured by people who overpromise. Not to "ruin the fun" but to instead make sure the party keeps going (sure, inviting a gorilla to the party may make it more exciting and "epic", but there's a good chance it also goes on a rampage and the party ends a lot sooner).

> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?

This is a very good distillation of one side of it.

What LLMs have taught us is a superficial grasp of language is good enough to reproduce a shocking proportion of what society has come to view as intelligent behaviors. i.e. it seems quite plausible a whole load of those people failing to grasp the point you are making are doing so because their internal models of the universe are closer to those of LLMs than you might want to think.

I think we already knew this though. Because the Turing test was passed by Eliza in the 1960's. PARRY was even better and not even a decade later. For some reason people still talk about Chess performance as if Deep Blue didn't demonstrate this. Hell, here's even Feynman talking about many of the same things we're discussing today, but this was in the 80's

https://www.youtube.com/watch?v=EKWGGDXe5MA

Ten years ago I was explaining to halls of appalled academic administrators that AI would be replacing them before a robot succeeds in sorting out their socks.

The field of AI needs to be constantly retaught the lesson that being able to replace important and powerful people doesn't mean your AI is actually intelligent. It means that important and powerful people were doing bullshit jobs.

ELIZA passed the Turing test in the same way a spooky halloween decoration can convince people that ghosts are real.

Eliza did not pass the Turing Test in any meaningful way. In fact, it did not pass it at all, and saying it did and comparing both is pretty disingenuous.

For real, even the 2023/2024 articles about chat bots passing the Turing test are pretty disingenuous. The research methods are extremely flawed.

....But this is falling into exactly the same trap: the idea that "some people don't engage the faculties their brains do/could (with education) possess" is equivalent to the LLMs that do not and cannot possess those faculties in the first place.

> What LLMs have taught us is a superficial grasp of language is good enough to reproduce a shocking proportion of what society has come to view as intelligent behaviors

I think that LLMs have shown that some fraction of human knowledge is encoded in the patterns of the words, and that by a "superficial grasp" of those words, you import a fairly impressive amount of knowledge without actually knowing anything. (And yes, I'm sure there are humans that do the same.)

But going from that to actually knowing what the words mean is a large jump, and I don't think LLMs are at all the right direction to jump in to get there. They need at least to be paired with something fundamentally different.

I think the linguists already knew this tbh and that's what Chomsky's commentary on LLMs was about. Though I wouldn't say we learned "nothing". Even confirmation is valuable in science

Scary thought

  > I have seen far, far too many people say

It is perplexing. I've jokingly called it "proof of intelligence by (self) incompetence".

I suspect that much of this is related to an overfitting of metrics within our own society. Such as leetcode or standardized exams. They're useful tools but only if you know what they actually measure and don't confuse the fact that they're a proxy.

I also have a hard time convincing people about the duck argument in [0].

Oddly enough, I have far more difficulties having these discussions with computer scientists. It's what I'm doing my PhD in (ABD) but my undergrad was physics. After teaching a bit I think in part it is because in the hard sciences these differences get drilled into you when you do labs. Not always, but much more often. I see less of this type of conversation in CS and data science programs, where there is often a belief that there is a well defined and precise answer (always seemed odd to me since there's many ways you can write the same algorithm).

> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?

I understand the difference, and sometimes that second statement really is true. But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one. I've talked to more than a few people who insist that an LLM can't have a world model, or a concept of truth, or any other abstract reasoning capability that isn't a native component of its architecture.

And I'm much less frustrated by people who are, in fact, claiming that LLMs can do these things, whether or not I agree with them. Frankly, while I have a basic understanding of the underlying technology, I'm not in the ML field myself, and can't claim to be enough of an expert to say with any real authority what an LLM could ever be able to do, just what the particular LLMs I've used or seen the detailed explanations of can do.

No; this is specifically about people who stipulate that the LLMs can't do these things, but still want to claim that they are or will become AGI, so they just basically say "well, humans can't really do it, can they? so LLMs don't need to do it either!"

I am an ML researcher, I don't think LLMs can reason, but similar to you I'm annoyed by people who say ML systems "will never" reason. This is a strong claim that needs be substantiated too! Just as the strong claim of LLMs reasoning needs strong evidence (which I've yet to see). It's subtle, but that matters and subtle things is why expertise is often required for many things. We don't have a proof of universal approximation in a meaningful sense with transformers (yes, I'm aware of that paper).

Fwiw, I'm never frustrated by people having opinions. We're human, we all do. But I'm deeply frustrated with how common it is to watch people with no expertise argue with those that do. It's one thing for LeCun to argue with Hinton, but it's another when Musk or some random anime profile picture person does. And it's weird that people take strong sides on discussions happening in the open. Opinions, totally fine. So are discussions. But it's when people assert correctness that it turns to look religious. And there's many that over inflate the knowledge that they have.

So what I'm saying is please keep this attitude. Skepticism and pushback are not problematic, they are tools that can be valuable to learn. The things you're skeptical about are good to be skeptical about. As much as I hate the AGI hype I'm also upset by the over correction many of my peers take. Neither is scientific.

  > But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one.

Requirement for proof is backwards. It's the ones that claim that thing reasons that needs proof. They've provided evidence (albeit shakey), but evidence isn't proof. So your reasoning is a bit off base (albeit understandable and logical) since evidence contrary to the claim isn't proof either. But the burden of proof isn't on the one countering the claim, it's on the one making the claim.

I need to make this extra clear because framing can make the direction of burden confusing. So using an obvious example: if I claim there's ghosts in my house (something millions of people believe and similarly claim) we generally do not dismiss someone who is skeptical of these claims and offers an alternative explanation (even when it isn't perfectly precise). Because the burden of proof is on the person making the stronger claim. Sure, there are people that will dismiss that too, but they want to believe in ghosts. So the question is if we want to believe in ghosts in the (shell) machine. It's very easy to be fooled, so we must keep our guard up. And we also shouldn't feel embarrassed when we've been tricked. It happens to everyone. Anyone that claims they've never been fooled is only telling you that they are skillful at fooling themselves. I for one did buy into AGI being close when GPT 3 came out. Most researchers I knew did too! But as we learned more about what was actually going on under the hood I think many of us changed our minds (just as we changed our minds after seeing GPT). Being able to change your mind is a good thing.

> "LLMs by their very nature cannot perform this task"

The issue is that it's not LLMs that can't perform a given task, but that computers already can. Counting the number of Rs in strawberry or comparing 9.11 to 9.7 is trivial for a regular computer program, but hard for an LLM due to the tokenization process. Where LLMs are a pile of matrixes and some math and some look up tables, it's easy to see that as the essential nature of LLMs, which is to say theres no thinking or reasoning happening because it's just a pile of math happening and it's just glorified auto-complete. Artificial things look a lot like the thing they resemble, but they also are artificial, and as such, are markedly different from the thing they resemble. is the very nature of an LLMs being a pile of math mean that it can not perform said task if given more math and more compute and more data? given enough compute, can we change that nature?

I make no prognostication as to whether or not AGI will come from transformers, and this is getting very philosophical, but I see it as irrelevant because I don't believe that AGI is the right measure.

>How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"?

Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place.

People just have their pre-conceptions about how they think LLMs should work and what their "very nature" should preclude and are so very confident about it.

People like that will say things like "LLM are always hallucinating. It doesn't know the difference between truth and fiction!" and feel like they've just said something profound about the "nature" of LLMs, all while being entirely wrong (no need to wait, plenty of different research to trash this particular take).

It's just very funny seeing people who were/would be gob smacked a few years ago talking about the "very nature" of LLMs. If you understood this nature so well, why didn't you all tell us about what it would be able to do years ago ?

ML is an alchemical science. The builders themselves don't understand the "very nature" of anything they're building, nevermind anyone else.

> Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place.

there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers.

> there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers.

fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large language modelling one. Those benchmarks show that those particular models have problems with accuracy on that many digits, not that LLMs fundamentally cannot be accurate on that many digits.

https://arxiv.org/abs/2405.17399

https://arxiv.org/abs/2307.03381

https://arxiv.org/abs/2310.02989

Ultimately, numbers are represented in GPT like systems in a pretty weird way. That way affects how well they can learn to do things like arithmetic and counting and the like but that way isn't a necessary way. You don't have to represent numbers like that.

> https://arxiv.org/abs/2405.17399

they built specialized model which is after bunch of trickery still has 99% accuracy(naive model had very low accuracy) on very simple deterministic algo. I also think most of the accuracy came from memorization of training set(model didn't provide intermediate results, and started failing significantly at sligtly larger input). In my book it is fundamental inability to learn and reproduce algorithm.

They also demonstrated that transformer can't learn sorting.

They did not build a specialized model. They changed the embeddings. You can do this for any existing model.

>They also demonstrated that transformer can't learn sorting.

They did not demonstrate anything of the sort.

What you are saying contradicts to my reading of publication.

Their method does not require building specialized models from scratch (you can but you don't have to) and they did not prove transformers can't learn sorting. If you think they did, then you don't understand what it means to prove something.

In my books what they built (specialized training data + specialized embeddings format) is exactly specialized model. You can disagree of course and say again that I don't understand something, but discussion will be over.

The data isn't specialized ?

A poor result when testing one model is not proof that the architecture behind the model is incapable of getting good results. It's just that simple. The same way seeing the OG GPT-3 fail at chess was not proof LLMs can't play chess.

This

>They also demonstrated that transformer can't learn sorting.

is just wrong. Nothing more to it.

>I also think most of the accuracy came from memorization of training set

Oh yes..it memorized a 20 digit training set to solve 100 digit problems. That makes sense. Lol

>(model didn't provide intermediate results, and started failing significantly at sligtly larger input).

No it didn't. They tested up to 100 digits with very high accuracy. I don't think you even read the abstract of this, nevermind the actual paper.

> No it didn't. They tested up to 100 digits with very high accuracy. I don't think you even read the abstract of this, nevermind the actual paper.

they have two OOD (out of distribution) accuracies in the paper: OOD: up to 100 digits, and 100+ OOD: 100-160 digits. 100+ OOD accuracy is significantly worse: around 30%.

If it walks like a duck, quacks like a duck, swims like a duck, and looks like a duck it's probably worth dissecting its internal organs to see if it might be related to a duck.

I think it will be an organoid brain bio-machine. We can already grow organs—just need to grow a brain and connect it to a machine.

The keyword being “just”.

Just grow, just connect, just sustain, just avoid the many pitfalls. Indeed just is key

  just adverb 
  to turn a complex thing into magic with a simple wave of the hands

  E.g. To turn lead into gold you _just_ need to remove 3 protons

You “just” need a more vivid imagination if that’s as far as your comment stretches.

I mean seriously, people on here. I’m just spitballing ideas not intended to write some kind of dissertation on brain-machine interaction.

That’s Elon musks department.

Okay. The keyword should have been “science fiction”

Somehow I doubt that organic cells (structures optimized for independent operation and reproduction, then adapted to work semi-cooperatively) resemble optimal compute fabric for cognition. By that same token I doubt that optimal compute fabric for cognition resembles GPUs or CPUs as we understand them today. I would expect whatever this efficient design is to be extremely unlikely to occur naturally, structurally, and involve some very exotic manufactured materials.

Maybe that'll be the first way, but there's nothing special about biology.

Remember, we don't have a rigorous definition of things like life, intelligence, and consciousness. We are narrowing it down and making progress, but we aren't there (some people confuse this with a "moving goalpost" but of course "it moves", because when we get closer we have better resolution as to what we're trying to figure out. It'd be a "moving goalpost" in the classic sense if we had a well defined definition and then updated in response to make something not work, specifically in a way that is inconsistent with the previous goalpost. As opposed to being more refined)

The something special about biology is it uses much less energy than a network of power-hungry graphics cards!

No one denies that. But there's no magic. The real baffling thing is that people refuse to pick up a neuroscience textbook

If a brain connected to a machine is "AGI" then we already have a billion AGIs at any given moment.

Well, I mean to say, not exactly human brains. Consider an extremely large brain modified to add/remove sections to increase its capabilities.

They already model neural networks on the human brain, even though they currently use orders of magnitude more energy.

But modifying a brain to be bigger and better doesn't require much in the way of computers, it's basically a separate topic.

Reply on news.ycombinator.com