That said all of the legal agreements become far more complicated and likely null and void if one person drops out or is added. I think an argument could be made, but with the complication you're basically saying polymarriages would be like corporations, with their associated complexity.
Polygamy as practiced now is a "bring more childbearing women" into the flock type thing, which is by nature non-equitable. I doubt most folks in a polygamous marriage (by choice) would want to get involved in a polyamorous marriage and as such, I think you can't legally support a non-equal union of that sort.