The thing you have to consider is genetics (parentage). Most don't men don't want to raise a child (and all the effort involved in that) that is not their own (without consent, outside adoption, etc). It's a very reasonable position and the basis for this arrangement appearing in most religions and legal systems.