The issue with scaling up data is that these models are already trained on effectively the entire internet. Additionally, humans learn to speak and understand the world with a tiny fraction of the language data given to LLMs. So, there’s pretty obviously a missing ingredient. I’m not convinced Marcus has the exact right answer, it’s actually somewhat difficult to suss out what he thinks should be done on a concrete practical level. That said, there’s a clearly a missing abstraction that allows much more efficient learning in humans.