Aren’t there efforts to simulate neurons? (Aka artificial brains)
in short, a continuously running graph that can rewire (instead of a feedforward network) with message passing; it would run training and inference simultaneously (inference would continuously retrain the network). Such a system running on the largest GPU clusters would be an interesting experiment in itself, although I don't think it would match the direct raw performance of today's AI models, and may never actually be better than just using feedforward networks trained by backpropagation, with access to external signals.
I'm also pretty sure people have been, and continue to, experiment with this. The current architecture is by no means the final form of LLMs. It's being tweaked continuously, though you don't hear about it as much anymore (unless it's DeepSeek dropping a bombshell every couple months).
Thing is, despite half the posters on HN and elsewhere arguing otherwise, LLMs are making such a massive, world-transforming progress, that humanity can't keep up with it. A good chunk of the world economy is now focused on AI R&D, and there's still not enough capacity to pick up all the low-hanging fruits found over those few years - there's still too much to gain from pushing the current paradigm.
Every now and then you see someone stumble over one of these low-hanging fruits, and it sparks a mini-revolution. Jev being the most recent example - a simple idea, arguably well-known in ML circles, but it's suddenly a big thing because someone stopped long enough to pick it up. There's many, many more such ideas still waiting for a minute of attention of someone with few million $ worth of GPUs to spare.