This is a common misconception. ChatGPT is not supposed to be good at this — it's a language model, not a maths model or data science model or whatnot. This is exactly why they have plugins, such as the one for Wolfram Alpha.
I don't think it really makes the case that you think it makes. Humans learn language and concepts through sentences, and in most cases semantic understanding can be built up just fine this way. It doesn't work quite the same way for math. When I look at the numbers in the example, I have no idea if they are prime or factors because they themselves don't have much semantic content. In order to understand whether they are those things or not actually requires to stop and perform some specific analysis on them learned through internalizing sets of rules that were acquired through a specialized learning process. Humans themselves don't learn math by just talking to one another about it, rather they actually have to do it in order to internalize it.
I suspect it's simply a case of we haven't sent ChatGPT to highschool yet. There will be some particular trick in terms of training methodology that trains the network to perform that more specialized analysis that doesn't simply emerge out of 'predict the next token from all this random internet text' as it's likely severely underrepresented in the training data in the first place, but I think it ought to be perfectly possible.
98.5% accuracy on addition arithmetic this way with GPT-3
No mathematician would make this exact mistake because it looks ridiculous, but for example I made similar mistakes multiple times, claiming low numbers prime or not prime wrongly during a reasoning. Especially when working at night.
There is a causality inversion here.
The only reason people know it can't do this is because they have tried and seen it cannot do this.
We do not have very precise bounds a-priori what GPT can and cannot do. We only learn them from black box testing.
That in no way precludes deriving deep understanding of some other concepts.
Even humans are not expected to be able to learn anything we throw at them.
https://ar5iv.labs.arxiv.org/html/2201.02177
This paper makes me believe it's less about the lack of ability to understand maths, but the power to learn more of it.
And this is the error I notice people very consistently making when they evaluate the intelligence of ChatGPT and similar models. They marvel at its ability to produce impressive truths, but think nothing of its complete inability to distinguish these truths from similar-sounding falsehoods.
This is another form of the rose-colored glasses, the confirmation bias we are seeing at the peak of the current hype cycle, reminiscent of when Blake Lemoine convinced himself that LaMDA was sentient. A decade ago, techies were dazzling executives with ML models that detected fraud or whatever as if by magic. But then, when the dazzling demos were tempered by the brutal objectivity and rigor of the precision-recall curve, a lot of these models didn't end up getting used in practice. Something similar will happen with ChatGPT. People will eventually have to admit what it is failing to do, and only then will we start up Gartner's fabled Slope of Enlightenment to the Plateau of Productivity.
RLHF and the tokenizer together explain many of the more common failure modes.
The way to test that, of course, is to give it problems that it hasn't seen. Unfortunately, because GPT has seen so much, giving it problems it definitely hasn't seen is itself now a hard problem. What's been shown so far: OpenAI's benchmarking is not always rigorous or appropriate, and GPT's performance is brittle and highly sensitive to the problem phrasing [1].
I agree with the article that GPT's training enables it to access meaningful conceptual abstractions. But there is clearly quite a lot that's still missing. For now, people are too excited to care. But when they try to deliver on their promises, the gaps will still be there. Hopefully at that point we will embark towards a new level of understanding.
[1] https://aisnakeoil.substack.com/p/gpt-4-and-professional-ben...
If a human mathematician said the things ChatGPT said in this dialogue, you would wonder if the person had recently suffered a severe traumatic brain injury.
If wrong answers still count because humans sometimes make mistakes, then I guess it won’t be too difficult to construct an impressive mathematical AI.
It's very tempting to give these systems the benefit of the doubt, but that tends to lead to hugely inflated conclusions about their capabilities. Remember that something as simple as ELIZA was perfectly capable of fooling humans who were predisposed to believe it was intelligent.
The system fundamentally cannot do this. You can make it generate text that is like what someone would say when asked to show their working, but that's a different thing.
> It's very tempting to give these systems the benefit of the doubt, but that tends to lead to hugely inflated conclusions about their capabilities.
I agree. I am seeing a bit too much over-optimistic predictions about these things. And many of these predictions are stated as fact.
Yes, we've had Wolfram Alpha for ages. For me, the biggest problem Wolfram Alpha is that it often doesn't understand the questions, and while I also sometimes get that with ChatGPT, the latter is much much better.
I've not had a chance to play with the plug-in that connects GPT to Wolfram Alpha.
> It's very tempting to give these systems the benefit of the doubt, but that tends to lead to hugely inflated conclusions about their capabilities.
This is an excellent and important point.
I think people who treat it as already being superhuman in the depth (not merely breadth) of each skill are nearly as wrong as those who treat it as merely a souped-up autocomplete.
I've only played with 3 and 3.5 so far, not 4, my impression is that it's somewhere between "a noob" and "genius with Alzheimer's".
A noob at everything at the same time, which is weird because asking a human in English to write JavaScript with all the comments in French and following that up with a request written in German for a description of Cartesian dualism written in Chinese is not something that any human would be expected to do well at, but it can do moderately well at.
Edit: I should probably explicitly add that by my usage of "noob", most people aren't even that good at most things. I might be able to say 你好 if you are willing to overlook a pronunciation so poor it could easily be 尼好 or 拟好 (both of which I only found out about by asking ChatGPT), so I'm sub-noob at Chinese.
I do think that this specific moment we're going through, in early 2023, is producing some of the most fascinating, confidently incorrect misunderstandings of chatgpt, and I hope that someone is going through these comment sections and collecting them so that we can remark on them 5 to 10 years down the line. I suspect that in time these misunderstandings are going to be comparable to how boomers misunderstand computers and the internet.
I recently asked ChatGPT to write me some Go code. What it wrote was mostly fine, but it incorrectly used a method invocation on an object, instead of the correct code which was to pass the object to a function at the package level (aka static method).
I think it's a real stretch to suggest that this could happen as a result of simply regurgitating strings it's seen before, because the string it emitted was not ever going to work. To me, it looked for all the world like the same kind of conceptual error that I have made in Go, and the only way I could see this working is if GPT had a (slightly faulty) model of the Go language, or of the particular (and lesser known) package I was asking about.
It's felt more like it "forgot" - like me - how the package worked, so it followed its model instead. That error was WAY more interesting than the code.
Or did you use a calculator to verify?
Because that would be exactly what ChatGPT with plugins does.