I am not applying any overly austere morals here. I think you would be hard-pressed to successfully argue that an affair in the traditional sense is not an immoral act. If you promise to be "faithful" to someone, and you break that promise and try to hide it, it is immoral based on most people's morals and understanding.
Sure, you might disagree with popular morals, that's fine. However, a relationship involves two people, and both should be valued equally. If one person thinks it is fine to break their promise and cheat in secret, so they do it despite knowing how the other person feels...they are putting themselves first. They are not treating both they and their partner as equals. That is the issue. It has nothing to do with some sort of ideological anti-affair "beliefs." It is simple respect.
I guess it is even technically "fine" to think you deserve more liberty than others, or that you think others' feelings don't matter as much as your desires (I'm not saying you do, I'm speaking generally). If that is the case, just own up to it and people can choose how to react accordingly. If you find partners that align with your views, great, everyone is on the same page, but if not, people deserve the truth. Everything else is up in the air, but they deserve to know. Keeping it from them is like robbing them of the choice to deal with it or leave.