Counting is a task that transformers can do, per Weiss.[1] But it's not surprising that transformer networks in general have trouble counting characters -- the tokenizer replaces common sub-strings, so the number of characters will not in general be the number of tokens. The network might have little way of even knowing how many characters are in a given token if that information isn't encountered elsewhere in training.
They will never achieve “reason” or understand what it means to do so; they are not human.
Sure, with enough input (in the form of LLM) it can predict what a human’s reasoning may look like, but philosophically, that’s a different thing.
Reason is not universal like how math is.
Maybe some red stuff comes out when you break it. Is it blood, or is it something pumped into the other side?
"A computer will never beat a chess master", etc.
Here are the facts for you: our reasoning is done by our brain. Our brain is just a bunch of processes. Those processes can be replicated in a computer. The number of cells and the speed can be improved. And there you have it, a superior reasoning machine.
These "only humans can do X" mostly comes from religion or other superiority bullshit, but in the end humans are not that special, although we seem to like to think so.
Superior? No.
Religion? No.
Philosophy? Yes.
Biologically, we're clearly an increment over the next smartest thing - we have the same kind of hardware, doing the same things, built by the same process. But that increment carried us through the threshold where our brains became powerful enough to break our species free of biological evolution, and subjecting us to much faster process of technological evolution. This is why chimpanzees live in zoos built by humans, and not the other way around.
If anything, biological history of humanity tells us LLMs may just as well be thinking and reasoning in the same sense we are. That's because evolution by natural selection is a dumb, greedy, local optimization process that cannot fixate anything that doesn't provide incremental benefits along the way. In other words, whatever makes our brains tick, it's something that must 1) start with simple structures, 2) be easy to just randomly stumble on, 3) scale far, and 4) be scalable along a path that delivers capability improvements at every step. Transformer models fit all four of the points.
> with enough input (in the form of LLM) it can predict what a human’s reasoning may look like, but philosophically, that’s a different thing
By what school of philosophy? The one I subscribe to (whatever it's name) says it's absolutely the same thing. It's in agreement with science on this one.
Computers are not biological. Therefore (imho) they will never obtain, and only replicate by trained example, the phenomenological human experiences.
So what? Biology isn't magic, it's nanotech. A lot of very tiny machines. It obeys the same rules as everything else in the universe.
More than that, the theoretical foundation on which our computers are built is universal - it doesn't depend on any physical material. We've made analog computers using water flowing between buckets. We've made digital computers from pebbles falling down and bouncing around what's effectively a vertical labyrinth. We've made digital computers out of gears. We've made digital computers out of pencil, paper, and a human with lots of patience.
Hell, we can make light compute by shining it at a block of plastic with funny etching. We can really make anything compute, it's not a difficult task.
We're using electricity, silicon wafers and nanoscale lithography because it's a process that's been best for us in practice, not because it's somehow special. We can absolutely make a digital computer out of anything biological. Grow cells into structures implementing logic gates for chemical signals? Easy. Make a CPU by wiring literal neurons and nerve cells extracted from some poor animal? Sure, it can be done. At which point, would you say, such a computing tapestry made of living things gain the capability of "phenomenological experiences"? Or, conversely, what makes human brains fundamentally different from a computer we'd build out of nerve cells?
I claim I can juggle. I pick up three tennis balls and juggle them. You hand me three basketballs. I try and fail. My original claim, that I can juggle, still stands.
The more correct claim is you can juggle [some small number of items with particular properties].
“Can you ride a bike?”
“Yeah.”
“Prove it. Here I have the world’s smallest bicycle.” <- this person is not worth your time and attention.
GPT is built on a computer. It is not a computer itself. I am made of cells, but I am not able to perform mitosis.
It’s also very capable of offloading counting tasks, since computers are so good at that. Just suggest it format things in code and evaluate it.
Here’s an example of it counting. I’m pretty confident it’s never seen this exact problem before.
How many fruits are in this list? banana, hammer, wrench, watermelon, screwdriver, pineapple, peach
There are 4 fruits in the list you provided: banana, watermelon, pineapple, and peach. The other items are tools, not fruits.
What does this mean? How can it possibly come up with the number 4 in a way that doesn’t involve counting?
There are two fruits in this list: banana and watermelon.
So what is the difference between GPT 4 and Chat GPT? Is GPT 4 all of a sudden running a counting algorithm over the appropriate words in the sentence because it understands that's what it needs to do to get the correct answer? Can anyone explain how you would get from whatever Chat GPT is doing to that by the changes to GPT 4? And more to the point, how does Chat GPT get to the answer 2 without counting? Apparently somehow, because if it can't even count to 4, I'm not sure how you could call whatever it's doing 'counting'.
This is begging the question.
> Can anyone explain how you would get from whatever Chat GPT is doing to that by the changes to GPT 4?
I don't know. I can't explain how a 2-year-old can't count but an 8-year-old can, either. 3.5 generally can't count[0]. 4 generally can.
You've really sidestepped the question. I suggest you seriously consider it. What's the difference between something that looks like it's counting versus actually counting when it comes to data it's never seen before?
Thanks for discussing this with me.
[0] Add "Solve this like George Polya" to any problem and it will do a better job. When I do that, it's able to get to 4.
> I can juggle
is here shorthand for
> I can juggle at all; I can juggle at least some things
and the basketball case is only a counterexample to the much stronger claim
> I can juggle anything
But the argument about AIs reasoning has little to do with such examples, because juggling is about the ability to complete the task alone. When it comes to reasoning there are questions about authenticity that don't have analogs I'm determine whether a person can juggle.
What would “not alone” mean? Do you think someone is passing it the answers? Of course it was trained, but that’s like cheating on a test by reading the material so you can keep a cheat sheet in your brain.
1) https://arxiv.org/abs/2305.00050.pdf 2) https://arxiv.org/pdf/1905.13211.pdf 3) https://arxiv.org/pdf/2203.15544.pdf
so it's great that it can reason better than humans on small-medium probems already well trained for, but so far Transformers are not reasoning (not doing causal graph analysis, or not even doing zero order logic), they are eerily well writing text that has the right keywords. and of course it's very powerful and probably will be useful for many applications.
I'm not quite sure what you mean that they cannot do causal graph analysis, since that was one of many different tasks provided in the various different types of reasoning studies in the paper I mentioned. In fact it may have been the best performing task. Perhaps try checking the paper again - it's quite a lot of experiments and text, so it's understandable to not ingest all of it quickly.
In addition, if you're interested in seeing further evidence of algorithmic reasoning capabilities occurring in transformers, Hattie Zhou has a good paper on that as well. https://arxiv.org/pdf/2211.09066.pdf
The story is really not shaping up to be 'stochastic parrots' if any real deep analysis is performed. The only way that I see someone could have such a conclusion is if they are not an expert in the field, and simply glance at the mechanics for a few seconds and try to ham handedly describe the system (hence the phrase: "it just predicts next token"). Of course, this is a bit harsh, and I don't mean to suggest that these systems are somehow performing similar brain-like reasoning mechanisms (whatever that may mean) etc, but stating that they cannot reason (when there is literature on the subject) because 'its just statistics' is definitely not accurate.
I mean the ANN in the inference stage when run does not draw up a nice graph, doesn't calculate weights, doesn't write down pretty little Bayesian formulas, it does whatever is encoder in the matrices-innerproduct-context.
And it's accurate in a lot of cases (because there's sufficient abstract similarity in the training data), and that's what I meant by "of course it'll likely be useful in many cases".
At least this is my current "understanding", I haven't had time to dig into the papers unfortunately. Thanks for the further recommendation!
What seems very much missing is characterizing the reasoning that is going on. Its limitations, functional dependencies, etc.
69, 50, 100, 70, 68, 60, 60, 60, 60 (GTP gathered its own guesses into this list for me BTW)
It seems if GTP is given "attention" enough, it can do the counting. But it cannot direct its attention freely, only as we give it instruction to do so.
I just did it again with 66 letter A's. Guesses were: 100, 100, 98, 67, 66, 66, 66, 66 -- GTP4 again settled on the correct answer. I also burned though my prompt quota for the next 3 hours :(
Also, as a GTP style challenge, how many numbers are in this message? You have half-a-second, go!
heavy-magpie|> I am feeling excited.
system=> History has been loaded.
pastel-mature-herring~> !calc how many Ns are in nnnnnnnnnnnnnnnnnnnn
heavy-magpie|> Writing code.
// filename: synth_num_ns.js
// version: 0.1.1
// description: calculate number of Ns
var num_ns = 'nnnnnnnnnnnnnnnnnnnn';
var num_Ns = num_ns.length;
Sidekick("There are " + num_Ns + " Ns in " + num_ns + ".");
heavy-magpie|> There are 20 Ns in nnnnnnnnnnnnnnnnnnnn.
This is why we see improvement in GPT when chain of thought/tree of thought is used with reasoning for each step. That can't correct every failure mode, but it increases the likelihood you'll receive a more correct answer.
The model outputs what it predicts a statistically normal output would fit in the context given.
Truly “llm” and these gpt tools are very much large scale “soundex” models.
Fantastic and great.
But not ai or even agi.
Is human intelligence anything more than a statistical model? Our entire biology is a massive gradient descent optimization system. Our brains are no different. The establishment of connectivity and potential and resistance, etc etc, it’s statistical in behavior all the way down. Our way of learning is what these models are built around, to the best of our ability. It’s not perfect but it’s a reasonable approximation.
Further it’s not soundex. I see the stochastic parrot argument too much and it’s annoying. Soundex is symbolic only. LLMs are also semantic. In fact the semantic nature is where their interesting properties emerge from. The “just a fancy Markov model” or “just a large scale soundex” misses the entire point of what they do. Yes they involve tokenizing and symbols and even conditional probability. But so does our intelligence. The neural net based attention to semantic structure is however not soundex of Markov model. It’s a genuine innovation and the properties that emerge are new.
But new doesn’t mean complete. To be complete you need to build an ensemble model integrating all the classical techniques of goal based agency, optimization, solvers, inductive/deductive reasoning systems, IR, etc etc in a feedback loop. The LLM provides an ability to reason abductively in an abstract semantic space and interpret inputs and draw conclusions classical AI is very bad at. The places where LLM fall down… well, classical AI really shine there. Why does it need to be able to do logic as well as a logical solver? We already have profoundly powerful logic systems. Why does it need to count? We already have things that count. What we did not have is what LLMs provide, and more specifically multimodal LLMs.
The algorithmic execution paper argues GPT 4 can do arithmetic woth 13 digit numbers before performance drops below 95%.