Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant I don't understand how you can have this point of view.
My PhD was applying simpler ML models to problems in structural biology (this was a while ago, so transformers and other large neural models didn't exist then), including protein structure prediction. Most recently I worked in applying generative models and neural network classifiers to problems in drug discovery. I have also worked in IR (search) and NLP, but again with older technologies.
The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap relative to those methods...which didn't work very well. The claims from laypeople that "protein folding is a solved problem" is total bunk, and yet we see that constantly. If you don't know the field, you will be fundamentally misled by statements like "breakthrough".
For other areas (e.g. NLP), the definition of "progress" is also well quantified, and is not accurately captured by the breathless hype that surrounds this technology. Are these large transformers a dramatic improvement? Absolutely. No question. Are they the end of white-collar labor? No. That's ridiculous. Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight.
> Idk what your "PhD" was in but everyone I know in the field is struggling to keep up with the pace of research so unless you are working on "Expert Systems" and trying to convince everyone that the past 10 years of DNN progress has been irrelevant
Except, I'm not trying to convince anyone of that. I'm saying that these models are evolutionary, progress in research is always a form of punctuated equilibrium, and picking random points of progress across fields and extrapolating to the infinite, glorious future doesn't work well. That's what people are doing today.
> Having worked in the space on-and-off since my PhD, the pace is not "overwhelming". You just started paying attention to it.
>ChatGPT and Stable Diffusion are good, but they're incremental gains over what came before, which were incremental gains over what came before that
>The AlphaFold example is apt: yes, it outperformed methods at CASP. It was indeed a dramatic leap
>Are these large transformers a dramatic improvement? Absolutely. No question.
>Has this been an evolutionary change over the last decade or so? Not a smooth curve, but yes. The world didn't suddenly change overnight.
Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.
No, I'm not. The field has been making gradual improvement for a long time, with transformers (ca 2017) being a notable jump forward. Research is always a form of punctuated equilibrium.
> Starting with AlexNet there have been numerous discontinuous improvements in CV, NLP, Game Playing, etc. It's just not true to say otherwise.
Nor did I. I said that these models have been advancing gradually for a long time. The fact that particular steps can be characterized as "jumps" when viewed in isolation is not a rebuttal.
But saying GPT-2 to 4 in 4 years isn't revolutionary just seems contrarian to me.
Bespoke NLP of any kind is out the door with a single model. That's fucking crazy.
This stuff is evolutionary.
[1] https://pub.towardsai.net/unified-language-model-pre-trainin...
This 'revolution' is quite fuelled by clever marketing and hype generated by Microsoft attempting to overthrow Google crown for better search. Unfortunately, that use-case has fallen flat on its face.
With hallucinations, bullshitting and regurgitating falsehoods and it continues to be unable to reason transparently and like all neural networks are just black-boxes with limited explanation of it's decisions, still requiring humans to review the output just like 'AI systems' still requiring humans to sit behind the wheel of a so-called robo-taxi 'AI' driver. Sounds like typical AI snake-oil.
The only 'safe' use-case for LLMs or even GPT-3 is 'summarization'. Everything else is complete hype; devoid of any reason to even justify the euphoric mania of one section of 'AI'.
A 'counter claim' without giving examples of refuting mine can be dismissed without any further explanation, since my point still remains un-refuted.
Summarization of existing text is the only safe killer use-case for LLMs than the rest of the so-called 'applications' and the hype of applying it to everything, where it makes little sense other than attempting to jump on another hype train.
I predict that the majority of these ChatGPT-enabled startups will be closing down in the long run, due to no product market fit, easily running out of money or another bigger company over-taking them.
What point, that LLMs are good at nothing other than summarization? That's not a claim arrived at via logic or evidence, it's a poorly thought out opinion that happens to be wrong. There's nothing to refute.