As mentioned that would be more like mixture-of-experts, where you instead effectively just multiply the rows with some of the columns in your weight matrices.
That said, I wrote my post after bed time and I'm now pretty sure I wasn't thinking straight.
Humans also only use X% of their brains (the one needed for a specific task)