In a perfect world I agree with you, but at some point you have to ask yourself why non-monogamy keeps going out of style after periods of acceptance. Even with a religion justifying it, Mormons gave up polygamy!

I think it's because polygamous relationships lend themselves to manipulative and abusive behavior more easily than monogamous relationships do. Over time that increase in probability of negative outcomes turns into social change and prescription against it.