Is this due to lack of human imagination?
So, yes, collective action is going to be a huge challenge for the foreseeable future, regardless of how much AI progresses.
Just shooting some thoughts based on it, as its kind of fun:
Several things are needed for AI to be used by everyone, including not having a Butlerian Jihad. Because it is a threat to many's present ways of working, many do avoid it intentionally or have an aversion to it.
Reasonable or not, such fears certainly prevent people from taking advantage of it.
But at a level of 'humanity', the issue is mainly due to reality being physical. It takes movement of time, energy, and matter for things to change at those levels because of the fundamental physics that powers reality.
That and Government.
The bureaucracy governments have is like matter itself: heavy, difficult to move, and sometimes a burden, slowing things down as well. It can prevent overshooting as a dampening force, but it can certainly hinder progress (looking at you, EU).
Because Markets can represent the flow of time, energy, and matter, the structures and constraints they have put in place may themselves directly redirect assets to goods that don't produce functional value (leverage). This would be like the vast majority of NFTs and crypto-casinos, and perhaps physics-doomed technologies like Star Cloud. I digress...
AI, even SI existing for different companies, may be optimal for them, but the market may easily not be. It won't allocate resources more effectively.
Streamlined coordination between multiple companies enabled by SI, maybe. (Anthropic anyone?).
But in total, I would say the bottleneck is not Human Imagination. It is:
* Matter, Physics, Energy. * Governments * Markets
Lack of imagination is also part of it I'm sure... looking at the silly things some people ask AI (tbh so do I though, out of curiosity) but failing to use it for what's truly important and where it can add value (this is what I try to pursue, mostly by just trying it a lot on anything, without stopping to think for myself).
On a side note, I really feel bad for Gen Z. Society has forced them to push AI as far as possible. They probably are the most aware of its limitations. Yet they are also the least able to criticize it without career repercussions.
I suspect this will change soon as more people come around to the fact that their chat bot is another tool just like Google search.
I used to think how wrong it was to compare an LLM to a search engine but have realized over time that despite being wrong, it's still a better mental model because it gets a fundamental detail right; LLMs are tools, they aren't agents, they aren't autonomous, they don't understand things, at least not in the way a human would understand them.
And this all ties back to the OP, that's the problem with calling them "smarter". Yes they can solve ancient math problems that humans failed at for many years and yet at the same time cannot be trusted to take a drive through order.
We need to stop seeing LLMs as a replacement for human cognition and instead appreciate what each form of "intelligence" brings to the table.
I was one of the very first users and developers of Bitcoin, April 2009 - Dec 2015. So for better or worse I did take part in something that changed the world. And here's the thing about that: you don't want lots of people changing the world. The world is as it is for reasons and many of the people who attempted to change the world ended up downgrading it, if not for themselves then for at least some other people. Textbook example: the long history of utopian socialist revolutions that always ended up bathed in blood.
There are relatively few world changes nowadays that everyone would agree on. That's why the utopian pitch for AI is always the same tiny handful of ideas, like curing cancer. Beyond basics like curing the last diseases, there aren't a whole lot of changes you can make that are obviously positive because most problems faced by modern societies are purely socio-political problems. They aren't problems that yield to application of intelligence. They're all about clashes of values and people struggling to impose their will on each other.
Of the few where it seems intelligence might make a difference, there are mundane reasons why it doesn't help much. Medicine is bottlenecked on clinical trials. They have far more ideas than they can check already, so generating even more won't help. Physics is bottlenecked on particle accelerator power. Ain't no LLM going to fix that. What might help is figuring out how those ideas would work out ahead of time, but that's hard because we can't simulate the body or the universe and LLMs don't have any magical solution to that either.
So the biggest impact of AI will continue to be in software, a domain where we actually are intelligence bottlenecked. LLMs can systematically drain the swampy backlogs of a gazillion badly maintained projects and that could probably absorb all the datacenter capacity being built today, if it were cheap enough. They can let us try out new platform architectures and then rewrite all the software that sits on top. Etc etc. And yeah right now some of that is limited by our imaginations and some of it is limited by cost, speed and other things that will hopefully continue to improve post-bubble.
We aren't solving shit even with agi. People are so far up their asses, it's unbelievable.
We have had the ability to feed the world for 50 years, the problem is not about quantity. It's distribution and unintended quantity, wicked problems, paradoxical results and simple cellular automate.
There's so many classes of problems that have zero algorithmic basis.