mixture of experts gates on the feed forward network only. the shared weights are the KQV projections for the attention mechanism of each layer.