I’m not deep into this story, but my initial reaction is that marital infidelity doesn’t seem like a strong reason to oust him, aside from him possibly being compromised by an international spy. Haven’t other successful leaders continued to lead after having their affairs exposed?
(And no, I don’t believe what he did is morally excusable for the record.)