It's complicated by the fact he is a military officer. Adultery especially by officers is seen as unbecoming conduct and a threat to military discipline.
Still, in my view, and I think in the view of many others, "flag" officers never really retire. And as such his behavior in this matter brings disgrace to the service and uniform. For me, it is just another recent example that really calls into question the ethics of an entire generation of our military. Patraeus, Ward, Sinclair, Allen, McChrystal, the list goes on and on.