Well that's just part of what you get into when you get involved with extramarital or otherwise clandestine sexual affairs.
It's legal, it's consensual, it's something you and your partners enjoy--and yet, you really would prefer that it not be Googleable during a board meeting.
Considering how puritanical and shamefully we handle talking about sexuality in the workplace now (despite claiming to support tolerance and diversity more than ever) it hardly seems surprising folks may still want privacy.
Or that you've paid for sex in Reno? Or that you've done heroin in portugal? All of these are not illegal in their respective areas and they're all socially frowned upon.
Are you not capable of empathizing with other people who don't do exactly what you do? Or are you just being pigheaded?