The downsides of being in a rigidly-defined monogamous relationship are all kind of obvious, I think. Most people do not experience love or attraction as zero-sum games: you can have a "crush" or whatever on Person B without diminishing your feelings for Person A. So a person in a monogamous relationship is going to miss out on some positive physical and emotional connections that might have been really enjoyable.
But...
I've known a fair number of people in poly/open/etc relationships over the years and they tend to be inherently unstable, even moreso than trad monogamy. Like you said, often one person wants more exclusivity.
Also... let's be totally honest. One partner is almost always going to have more access to sex and love outside the relationship. Either they are more attractive, more assertive, or simply have more free time, or any other number of reasons. So the "openness" never seems to work out in a totally equal and/or equitable way.
They also seem to run into the problem of time and energy. In the abstract, love and sex are not zero-sum games. But a person only has so much energy and so many free hours in a week. So in practicality, yeah. It does become a bit zero-sum.