Turns out GPT-4 was 8 kids in a trench coat. And hey, why not? If the brain can have regions, perhaps that's a likewise a useful abstraction for LLMs. I'd love more info on model performance differences between monolithic vs manifold/subdivided architectures. Does model composition give us a "more than the sum of its p̶a̶r̶t̶s̶ weights" phenomenon, or just some implementation conveniences?
Side note: I loved the George Hotz interview on Lex Fridman's podcast. I strongly disagree with some of his opinions, but thoroughly enjoyed hearing them. I highly recommend it, if you're looking for some entertainment.