It seems like it is getting tripped up on grammar. Do these models not deterministically preparse text input into a logical notation?
I was quite amazed that during 2014-2016, what was being done with dependency parsers, part-of-speech taggers, named entity recognizers, with very sophisticated methods (graphical models, regret minimizing policy learners, etc.) became fully obsolete for natural language processing. There was this period of sprinkling some hidden-markov-model/conditional-random-field on top of neural networks but even that disappeared very quickly.
There's no language modeling. Pure gradient descent into language comprehension.
> itdoesntwork.jpg
Grammar isn't how language works, it's just useful fiction.
It seems that every time we try it, we find out that when model picks up the language structure on its own, it ends up being better at it than if we try to use our own understanding of language as a basis. Which does seem to imply that our own understanding is still rather limited and is not a very accurate model.
On the other hand, the fact that models get amazing translation capabilities just from training on different languages (seriously, if you are doing any kind of automated translation, do yourself a favor and try GPT-4) implies that there is a "there" there and the Universal Grammar people are probably correct. We just haven't figured out the specifics. Perhaps we will by doing "brain surgery" on those models, eventually.
For example, machine translation attempts were morphing the parse trees , document summarization was pruning the grammar trees etc.
I don’t know what your high level task is, but if it’s just collecting names then I can see how a specialized system works well. Although, the underlying model for this can also be a NN, having something like HMM or CRF turned out to be unnecessary.
On the other hand, people seem to be using GPT-4 for simple text classification and entity extraction tasks that even a small BERT could do well at a fraction of the cost.
Given a string of text, what's the most likely text to come next.
You /could/ rewrite input text to be more logical, but what you'd actually want to do is rewrite input text to be the text most likely to come immediately before a right answer if the right answer were in print.
* Unless you mean inside the model itself. For that, we're still learning what they're doing.
Phrasing it like that, it sounds like the stack has become analog -> digital -> analog, in a way…
I think this is a perfect example of why these things are confusing for people. People assume there's some level of "intelligence" in them, but they're just extremely advanced "forecasting" tools.
That said, newer models get some smarts where they can output "hidden" python code which will get run, and the result will get injecting into the response (eg. for graphs, math, web lookups, etc).
This "LLMs are just fancy autocomplete so they're not intelligent" is just as bad an argument as saying "LLMs communicate with text instead of making noises by flapping their tongues so they're not intelligent". Sufficiently advanced autocomplete is indistinguishable from intelligence.
A smart entity being able to emulate a dumber entity doesn't support in any way that the dumber entity is also smart.
That doesn't mean LLMs are intelligent, just that you can't claim they're unintelligent just because they generate one character at a time.
From my point of view as someone reading the comment I can't tell if it's written by an LLM or not, so I can't use that to conclude if you're intelligent or not.
From my point of view as someone reading the comment I can't tell if it's written by an LLM or not, so I can't use that to conclude if you're intelligent or not."
There is no scientific evidence that LLMs are a close approximation to the human brain in any literal sense. It is uncouth to critique people on the basis of what appears to be nothing more than an analogy.
Since we don't really understand the brain that well that's not surprising
I never said that, just that as a black box system that generates words it doesn't matter if it's similar or not.
> People assume there's some level of "intelligence" in them, but they're just extremely advanced "forecasting" tools.
My question wasn't meant as a quip. Rather it was literal-- how do you know your intelligence capabilities aren't "just extremely advanced forecasting"? We don't know for sure, and the answer is far from obvious. That doesn't mean humans are advanced LLMs-- we feel emotions, for instance. My comment was restricted to intelligence specifically.