LLMs copy paste from their training set. That's plainly obvious when you try to do real work with them.
It feels like language models are becoming more capable of picking novel problems apart, identifying strategies for the pieces, and then executing the pieces to come up with a complete solution. This is part of what makes CoT reasoning pay off so well.
I am interested to see what comes after CoT reasoning (perhaps the latent-space reasoning [0] that recently hit the front page?), and how it'll affect which problems can be effectively solved with a model. But I feel like it's not very useful to say models completely fall apart on novel tasks.
Maybe they fall apart when the solution includes completely novel strategies, but sometimes models engineer new strategies too, so even that's not completely off the table.
But then again, you could still say that if it's possible to just engineer a solution out of the blue by recycling patterns from the training data, was the problem ever even novel in the first place?
Also consider that models don't only directly learn patterns from the training data. There's a limited degree of generalization that enables the model to reproduce the patterns of the training data, but not necessarily only the patterns of the training data. The way loss is modeled in training is designed to allow the model to plausibly output the training data, not to ensure the model outputs nothing else.
That's a very polite way to say that it's complete bullshit. You can't really have a discussion with people who aren't engaging with what the technology actually does, but rather with their biased, quasi-religious belief about what it can't possibly do - namely, compete with humans on intellectual tasks. When the reality is that we're all outclassed by these models, in many respects.
It's not the worst rule of thumb I've seen, and it does explain some things. It's just not entirely accurate, as I said. There's some nuance it doesn't capture, but it does capture some of the picture.
"Complete bullshit" would be more than a somewhat-accurate, yet incomplete understanding to me. It would be... well, complete bullshit. And I would probably call it such.
LLMs do fantastic on problems they can copy out of their training set, but they fall apart on actually novel tasks.
You can make the statement true if you raise the bar for "novel task" to a point where most humans also can't do them. In these threads the goal posts for words like "understanding", "creative", "novel" etc gets moved for LLMs to way past their previous definitions, in fact way past a large % of the worlds human software developers.
To just take one example: Ask for a security review and you get back a list of all the things you would have found yourself after some days of pouring over the code (if you are senior developer well versed in security issues), plus the ones you would have overlooked, but in 15 minutes instead of in several days.
Before LLMs came along this would obviously have been "novel work". A shining example of a human using their intellect! But now that LLMs can do it it is just "copy and paste work"...
FRACTRAN. Given 2^x return 3^(x^3).
https://chatgpt.com/share/6ad58a36-5278-83ea-b1f5-22534915e1...
I entered two prompts to it:
Describe the language FRACTRAN.
Write a FRACTRAN program that takes 2^x and finishes with the value of in the 3^y of y = x^3.
4 (2^2) -> 6561 (3^8). Other values may be in other power registers.
The program that it came up with was:
38/17 1495/38 29/19 19/23 62/377 37/29 29/31 41/185 89/37 3913/82 47/41 41/43 106/611 59/47 47/53 61/413 37/59 9581/122 71/61 61/67 146/923 79/71 71/73 249/869 59/79 79/83 97/178 1/89 89/97 17/2
You can paste that into https://tjwei.github.io/Fractran/ and run it. Put the input of 2^3 and I recommend a 1000x speedup. It's output was 3^27
---
One can certainly go adversarial with their tasks... though I know junior programmers that fall apart on things they can't look up on Stack Overflow either.
For tasks that I've worked with it on, I've been reasonably impressed by it. Though I'll also note that a lot of the work that we do isn't new.
There are register machines out there. When I search google for "register machine cube of number" it comes up with one. The question of how do you apply the register machine model for that (if you're claiming that is in its training data rather than searching for it like I did) and then compile that into FRACTRAN is about chaining together tools reasonably... it appears to have written a compiler in Python for a model register machine to FRACTRAN.
Lets ask a different question... if I gave you the task of writing that program in FRACTRAN, would you attempt to solve it in the same way?
LLMs are clearly able to apply seemingly arbitrary transformations to symbol streams in ways that generalize well beyond the training set. It's getting into metaphysics whether that is "novel" but it's definitely not "copy paste from the training set"
You may agree with the author (that it doesn't mean it understands the language) but it's pretty hard to argue that this is copied form the training set.
The code it produces is a clear copy of the spec or the existing code that is fed to it as part of the task. It is just following the spec instructions, and some models follow it so closely that they get caught out by the lies/adversarial traps in the spec. Opus 5 is better at seeing through the lies, but it still very much copies from its context window.
Humans look at the docs too! Is there any difference?
What's plainly obvious is that you've never tried to do real work with them.
It's easy to tell when you're dealing with posers: they describe their brief aborted experience from a year or more ago as though it's universal.
What sticks out is really the lack of people doing something totally new. Look back on history and it seems like humans can't do anything except as in response to some input that we re-mix. (Although it is across fields -- a change in economic conditions or scientific outlook may affect music and painting; a small improvement in factory production methods may ripple down to experiments validating general relativity, etc)
Why didn't Mozart create a piece of jazz or rock to mix things up? Why are there no medieval cubistic painters? Why didn't someone understand $scientific_topic before?
Most such questions can be answered by some kind of "the social environment and culture at that time didn't prepare people to do it". I.e., humans couldn't remix it yet from what was already there.
Yes there are leaps but they seem to always be within some small "box" that prior knowledge creates for us. We can jump one inch at the time but no further..
Do you believe humans are any different, in general? What "actually novel task" have you recently aced?
> LLMs copy paste from their training set.
This doesn't follow, and isn't consistent with even the most shallow understanding of how a neural network works.
If we critique your own thought process here in the same way, we'd have to conclude that you're not a very useful model.