An ensemble is basically a mix of models for different tasks. One model could be an LLM, another image understanding model etc. Different tasks could be passed to different models or every task could be passed to all models and a result collated etc.

MoE is...well basically a way to have a large model without computing all the parameters at once. So you take several smaller language models and you train them all on subsets of the same dataset. Then you train them to make predictions together. You could train for switching experts at the token level i.e one expert picks one token and another picks the next etc

The "experts" are not clearly delineated or known. One "expert" could be a capital letter expert etc.

People see GPT-4 being MoE and they go "Oh so questions about medicine are being passed to a separate model than questions about say Mathematics etc" but that's a misconception.