> you get severe slowdowns when weights are that large.

Not necessarily for MoE