I don't care that he had an affair (none of my business; who knows the terms of his relationship with his wife; Americans / humans are far too moralizing in controlling sex) and, even if it was wrong in the context of his marriage, I don't think someone should be fired for that.

If he knew / suspected she was a spy, yes, absolutely. If she was given access to confidential information, yes, absolutely.

If he thought it was a romantic relationship, fine, that doesn't affect me / NVDA / America.

Obviously he should stop now that he knows! But if he didn't at the time, you know, I think I can still trust him to make computer chips.