Symbolic AI has died a miserable death, and he never recovered from it.
Symbolic AI has died a miserable death, and he never recovered from it.
46:12
> What are you excited about in 2025? What's to come?
> AGI. Excited for that.
Of all the possible defenses of this remark, I didn't expect that one.
Besides all that, think about it: if this wasn’t a joke, then why has he never said the same thing again?
> OpenAI CEO Sam Altman wrote in January: “We are now confident we know how to build AGI.” This is after he told a Y Combinator vodcast in late 2024 that AGI might be achieved in 2025 and tweeted in 2024 that OpenAI had “AGI achieved internally.”
https://fortune.com/2025/08/25/tech-agi-hype-vibe-shift-supe...
Oh it’s all just jokes though, jokes that Sam has profited enormously from.
>... from here to building an AGI will still take a huge amount of work there are some known unknowns but I think we basically know what to go what to go do and it'll take a while, it'll be hard but that's tremendously exciting... (38:59)
which to me doesn't sound like we'll be done within a year.
But we live in the era of Elon Musk, so what the hell. I'm going turn my belly button fluff into infinite gold next year. Can I have a few billion?
It’s also worth noting that even when he does talk about AGI, he makes a strong distinction between AGI (human level) and ASI (super human level) intelligence. Many people in these kind of discussions seem to conflate those two as being the same.
I know I for one was shocked to take my first AI course in undergrad and discover that it was mostly graph search algorithms… To say the least, those are still helpful in systems built around LLMs.
Which, of course, is what makes Mr. Marcus so painfully wrong!
Symbolic regression has an extremely obvious advantage over neural networks, which is that it learns parameters and architecture simultaneously. Having the correct architecture means that the generalization power is greater and that the cost of evaluation due to redundant parameters is lower. The crippling downside is that the search space is so vast that it is only applicable to toy problems.
But Gary Marcus is in favour of hybrid architectures, so what would that look like? In the SymbolNet paper, they have essentially decided to keep the overall neural network architecture, but replaced the activation functions with functions that take multiple inputs aka symbols. The network can then be pruned down into a symbolic expression.
That in itself is actually a pretty damning blow to Gary Marcus, because now you have most of the benefits of symbolic AI with only a tiny vestige of investment into it.
What this tells us is that fixed activation functions with a single input appear to be a no go, but nobody ever said that biological neurons implement sigmoid, relu, etc in the first place. It's possible that the spike encoding already acts as a mini symbolic regressor and gives each neuron its own activation function equivalent.
The "Neuronal arithmetic" paper has shown that biological neurons can not only calculate sums (ANNs can do this), but also multiply their inputs, which is something the activation functions in artificial neural networks cannot do. LLMs use gating in their MLPs and attention to explicitly model multiplication.
There is also the fact that biological neurons form loops. The memory cells in LSTMs perform a similar function, but in the brain there can be memory cells everywhere whereas in a fixed architecture like LSTMs they are only where the designer put them.
It seems as if the problem with neural networks is that they're too static and inflexible and contain too much influence from human designers.
>Gary Marcus tried to tell you for years that this moment would come for years. Do consider reading his 2020 article...
I can't say I agree with the sentiment "Game over". The game of trying to develop AI is very much on.