It's very likely GPT-4 is an ensemble. A single model won't be able to keep up, even with this level of parameters.
Run a fleet of these together, however...
Run a fleet of these together, however...
MoE is...well basically a way to have a large model without computing all the parameters at once. So you take several smaller language models and you train them all on subsets of the same dataset. Then you train them to make predictions together. You could train for switching experts at the token level i.e one expert picks one token and another picks the next etc
The "experts" are not clearly delineated or known. One "expert" could be a capital letter expert etc.
People see GPT-4 being MoE and they go "Oh so questions about medicine are being passed to a separate model than questions about say Mathematics etc" but that's a misconception.
I assume MoE is still prohibitively expensive to train, which is why we're not seeing massive MoE models?
Only recently has it looked like we could get equitable performance from MoE architectures.
https://arxiv.org/abs/2305.14705
https://arxiv.org/abs/2308.00951
In the first paper, you can see the underperformance i'm talking about.
Flan-Moe-32b(259b total) scores 25.5% on MMLU pre Instruct tuning and 65.4 after.
Flan 62b scores 55% before Instruct tuning and 59% after.