You make good points, but this one isn’t quite what I meant. I didn’t mean we are trained by evolution for a particular language, but that evolution selected for a language skill. In other words, we are “pre-baked” with the ability to learn a language. It would be analogous to a DL model being trained for generalized regression but not a particular problem. To that extent, I think we’re saying the same thing. Some of the theories related to our generalized learning abilities postulate they stem from this base ability (our aptitude for music, for example, being a consequence of our language learning ability)
>I see BERT as non sensical. You need to be scientific and have a mathematical theory
This is a matter of contention. A lot of science progresses by starting with an empirical result that drives a change in theory rather than the other way around. I can’t remember who to attribute it to, but there’s a quote to the effect that “‘Thats odd’ is the most productive phrase in science, rather than ‘Eureka!’”
Yes it did, and we do know a lot about general principles behind, from different disciplines. This is still an early science. NLP research still hasn’t considered many important aspects of language learning that we have discovered in such a short period of time.
> starting with an empirical result
What makes you think these benchmarks are empirical? They were hand constructed to fit some objective, assuming that being good at the said objective is required for NLP tasks. Where’s the empirical experiments to validate the notion that said objectives lead to language? Science hasn’t worked this way, datasets aren’t constructed, you do an experiment and MEASURE it. Then you make models, try to explain the phenomenon, and test new ideas and validate your models. Can your model extrapolate new information and suggest new experiments to validate? I used the word extrapolate over predict intentionally.
I assume the empirical results are the validation sets run. I.e., the tests that show it provides better results than the base rate. Again, it’s important not to conflate verifiable results with understanding the underpinnings of why it works. If you’re walking and I race you with my car over and over again, I can conclude my car is a faster mode off transportation without understanding anything other than “push right peddle to go faster”. My ignorance doesn’t invalidate the results
And it's not the whole picture to say the brain only uses 15 watts - when there's all sorts of necessary support systems that it couldn't run without. So it's closer to 100 watts (2000kcal/day)
All in you require about 100W (2000kcal/day), maybe 3 times that when doing a lot of physical activity. Boston Dynamic's Spot uses about 400W. I can probably outperform it in some disciplines while it would beat me in some others. That would be a fair comparison.
You’re making sweeping assertions that require domain experts in several different disciplines. The scientific evidence points to the contrary. It is demonstrated that humans have a general ability to learn wide range of things, without being genetically programmed to. None of us evolved to drive cars. Yet nearly everyone in my grandparents generation were able to learn this totally new skill despite being a new invention, where you couldn’t possibly have had time for evolution to act. They weren’t genetically evolved to drive, it was learned within their lifetime and generation. There are tribal humans in different parts of the world that haven’t developed written language. Yet you can teach them written language. Where’s your “pre trained brain” theory there?
That is if we don't kill ourselves by making stupid mistakes.
I think 30 years is probably reasonable, if for “30 years” one reads “twice as long from now as commercially viable fusion generation actually was when first widely hailed as 15 years away”.
Theory is required for understanding, not for discovery. They are two different things. What gets a lot of scientists and statisticians worked up is how ML can outperform traditional models without always being interpretable. It’s like claiming the Wright brothers couldn’t build a flying machine without having a thorough understanding of the mechanics of flight. I can improve performance of my car by remapping the fuel without necessarily understanding the nuances of optimizing enthalpy. There’s levels of understanding, and they may not correlate completely to effectiveness. To that extent, it’s more engineering than science