Whenever we discover how the human mind does it, and then we apply it to AI systems. Currently AI just has a surface level interface that makes it seem like it can think of all these things humans think about, i.e. is this going to work properly, is this a good idea given the objectives, does this achieve the wider goal etc., etc.

It's ultimately hard to articulate and hard to measure. We are only measuring very narrow tasks (even if the tasks are large, there is no test that is measuring how well the AI took initiative or was able to solve unforeseen problems).

People like Yann LeCun understand this and are trying to build more wholistic level intelligence that would achieve this, while the majority of the current AI industry is closing their eyes and betting on scale and hoping that with enough compute these properties will emerge. I think we're going to be stuck with "dumb intelligence" until those other breakthroughs happen.