It seems consistent with the behavior we see when using GPT4 in chat mode. Every once in a while it will change its answer as it’s generating it, as though it’s switched which model it favors to produce the response. GPT3.5 doesn’t do that.
I agree with you that whole-output MoE models don’t.