Further improvements in efficiency need not come from alternative architectures. They'll likely also come from novel training objectives, optimizers, data augmentations, etc.
My bet is on sparsity, lottery tickets and symmetries.
While these things represent a fundamental way we store information as humans, these have very little to do with actual reasoning.
My bet is that Hebbian learning is going to see a resurgence. Basically the architecture needs to be able to partition data domains while drawing connections between them, and being able to run internal prediction mechanisms.
However, I don't think Hebbian learning will see a resurgence except maybe if it motivates some kind of pruning mechanism.
I think that Sutton was right in 'The bitter lesson', the problem seems to be that we are hitting the limits of what we can do with our compute.