> at least provide evidence that scaling up no longer works for transformers.
Isnt the Mixture-of-Experts trend (GPT4 is MoE?) kinda of a proof ?
Isnt the Mixture-of-Experts trend (GPT4 is MoE?) kinda of a proof ?
> I said “scale is all you need for transformers”
No you didn't. What kicked this all off was
> What’s the main difference between an ape’s brain and a human brain? Scale.
Don't retcon.