Working with these models every day, it's clear that they can certainly interpolate between points in latent space and generate sensible answers to unseen questions, but it's pretty clear that they don't generalize. I've seen far to many examples of models failing to display any sense of generalization to believe otherwise.
That's not to say that interpolation in a rich latent space of text isn't very useful. But it's not the same level of abstraction that comes from true generalization in this space.
Failing to generalize on whatever you have in mind is not evidence that the models are incapable of generalization. If you really think so, just be prepared to write off a good chunk of humans as well lol.
This is generalization. https://general-pattern-machines.github.io/
Just seems to me like you've taken examples of generalization and invented an alternate meaning just so you can't admit it generalizes.
"Oh but you see this example of answering an unseen question is "insert meaningless distinction" so it doesn't really count"
The one possible exception is logical inference, and this problem seems tractable with tool use or programming.
-ChatGPT initially gives the same diagnosis the vets did, Babesiosis
-It notes that Babesiosis may have either been a misdiagnosis or there may be a secondary condition/infection causing the remaining symptoms after the Babesiosis treatment didn't resolve all of them
-It suggests such a hypothetical secondary condition could be IMHA, which the article notes is an extremely common complication of Babesiosis with this specific dog breed
-A quick Google search brings up a fair amount of literature about the association between Babesiosis and IMHA
So in fact this is the opposite of a never before seen situation, ChatGPT was just regurgitating common comorbidities of Babesiosis and the vets in question are terrible at their job.
If it was 4 I’d be curious about the specific problem if you’d be willing to link to the chat.
Gpt4 only ever suggested grammar validator to the point I had given up and was going to write a grammar generator, and so I started looking for the equivalent of antlr in python, and in three searches I find out nltk.grammar that actually solves the original problem
It's not a new library either, so I'm dumbfounded by why gpt4 couldn't make sense of my request.
https://chat.openai.com/share/54cb8876-96ff-4434-a479-4d2dde...
No terminal matches 'd' in the current parser context, at line 1 col 5
cat do ^ Expected one of: * CAT * SPACE * FISH * DOG
Next valid symbols: ['* CAT\n\t* SPACE\n\t* FISH\n\t* DOG']
Or given the context space is a valid production from the last error, but then I would need to roll back the incomplete "do" terminal and lose that context.
Yeah, it's telling you what it was expecting when it hit an invalid token.
You take the invalid part and filter the list of suggestions with it. If you tell chatgpt about the problem it solves it.
(Edit - when I was running this, having any number of spaces was parsed as valid btw if that's the concern - space was a valid option)
IMHO OP is talking about "explainability" of the results, which is notoriously bad for current AI. For certain applications (idk if SQL would be one but mortgage application might be one) it is required to be able to explain how the computer got to the decision.
Binary classifiers don't generalize?
Just because my output is not generative does not mean we are cannot learn / generalize elsewhere. Think of it as a 2-stage process.