If you look at the history of espionage going back n thousand years you'll see plenty of cases where people have been blackmailed, extorted, or otherwise compromised through illicit romantic affairs.

It may not be such a big deal if you're a four-star general but it's a very big deal in the intelligence community. If this sort of thing was tolerated at the top level of the CIA, the agency would have no moral authority to enforce the rules among the rank-and-file cloak-and-dagger types.

OTOH if he had reported the affair(s) in question as he was supposed to do, there might have been some room for negotiation. It's not so much the affair, but the cover-up.