It’s interesting that humans have decided to allow violence to be acceptable and commonplace, celebrated even, while sex is the thing that is taboo. Seems backwards, especially when you consider consent.
Some people don't consent to infidelity. It's very practical.
People who are shopping around for sex are rarely reliable and committed and you couldn't trust them at their word if they say otherwise.
If it's consented to, it's not infidelity.
The thing you have to consider is genetics (parentage). Most don't men don't want to raise a child (and all the effort involved in that) that is not their own (without consent, outside adoption, etc). It's a very reasonable position and the basis for this arrangement appearing in most religions and legal systems.