There is a deal of sex-based difference here: being an older mother is harmful for the mother and the child; being an older father is far, far, less so (older fathers might be better for their kids?). Young women can choose a mate from all ages of men. Young men need to choose a mate -- if they wish to procreate I mean -- from the pool of fertile and healthy younger women. Medical science has eroded a little of that difference, but it still starts to get dangerous for women to fall pregnant in their late-30s.
An example of how that difference plays out is that in a childless couple at age 50 the woman basically has no chance to have children of her own, but the male could quite reasonably be expected to be fertile for a couple more decades. Thus, if a couple are unable to procreate and the male wishes to do so, then there is a tension; a biological impetus towards them leaving their wife and finding another partner.