Hard to overstate how exciting I find the fields of hypernetworks and metalearning. For good or for evil, this I feel is almost certainly how we create AGI. GNNs make perfect sense as a method of accomplishing it too -- after all, neural networks are just graphs anyways.

I don't believe AGI is going to conform to our rigid, symmetrical, boxy human-made architectures with their nice, neat flow patterns and powers-of-two factors everywhere -- those architectures work great for what they're designed to do, but achieving sentience/true intelligence just really doesn't feel like it's going to come out of a feedforward MLP, or even a transformer (in my opinion).

The most complicated and capable neural networks we know of are the organic, messy, squishy, yet organized electric meat lumps in our own skulls, produced by billions of years of being exposed to the same loss function over and over and over, but with unlimited ways to adapt in order to minimize that loss function. Nature makes awesome, fractally, insanely complex networks that we could never design by hand. I think it's only sensible to assume that AGI will work similarly.

Of course, this is all just armchair speculation. I'm not an expert, just an enthusiastic data scientist who thinks these next twenty years in AI are going to be a heck of a freaking ride.