I can see why Wolfram is bearish on what might be termed "the naive scaling hypothesis", i.e. that given more data, LLMs will naturally cease making false utterances by learning more systems of knowledge. If the naive scaling hypothesis is true, it recapitulates and invalidates a good chunk of the hand-coded work that he's built over the last decade or two. But I am not so sure; my money is on the robots for now.
For example, it should be really easy to generate training data for a LLM using more-formal systems like Wolfram Alpha; not least by having your LLM generate an arbitrary large list of "questions for Wolfram Alpha", then take that query and put it into WA, then attach the results to your LLM training set.
In other words, systems like Wolfram Alpha will be used to boost LLMs; the other way round is less obvious. Given the recent success, I'd put my money on "LLM can learn any structured system that can be wrapped in a text interface".
An example that's even more impressive than "wrap Wolfram Alpha" has already been demonstrated: LLMs plugged into Physics models (MuJoCo) at Google (https://arxiv.org/abs/2210.05359). There is (currently) no reason that these models can't be plugged in to learn any given simulator or oracle.
And on a more prosaic note, Google's LaMDA is clearly ahead of ChatGPT (it's just not public), and explicitly tackles the bullshit/falsehood problem by having a second layer that fact-checks the LLM by querying a fact database / knowledge-graph.
Of course, perhaps at some point before AGI the "naive scaling" approach will break down. It just seems to be a bad bet to be making right now; we are seeing no evidence of a slowdown in capabilities gains (quite the opposite, if anything).