ashwon13

This is less so a hedge against Chinese labs catching up to the frontier/OAI Anthropic falling in valuation, and more a hedge against a complete recession. We are overleveraged in AI investment and if tomorrow any Chinese Lab can drop a model at the frontier what was the point of all those billions. Now, if you can protect the American model from the bubble popping even for just long enough to hedge risk a little better, then you may just prevent collapse. The problem is though, that the market is heavily irrational right now and if anything this may just be the signal to investors to double down on American AI.

Would like more details technically on their approach. Is it a neuro-symbolic approach, do they do something beyond just scaling, or some other architecture breakthrough, but alas drug discovery must remain proprietary. Thank gods for capitalism!

I think he argues more that LLMs are a form of supervision because you need a whole batch of text about specifically what you want the LLM to learn for it to be useful at that one topic. That they predict on human supplied data instead of learning purely from existing in the world and from observations.

I think you misunderstand what he thinks LLMs can and can't do. He says repeatedly that LLMs may be great for generating text and code, but that a fundamental model of artificial intelligence should be able to perceive and use information outside of text. Also, that a supervised learning approach is not exactly representative for how we learn. That it's a small piece but we largely learn with unsupervised learning. His main criticisms of LLMs are that they are supervised, probabilistic, and learn largely from text instead of observations. His claims about performance come downstream and I'd still argue that he's been (somewhat) right about those as well.

That LLMs don't have common sense and don't have good physical reasoning abilities; that you can't scale LLMs all the way to AGI; or that they can't predict the consequences of their actions which is the foundation of agentic behavior all seem like still (mostly) accurate predictions to me.

While LeCun has his share of problems, I think largely his criticisms on LLMs are more right than wrong. What remains to see is how good JEPA can be at filling in the gaps left behind from the brittleness of LLMS.