You could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.