This means the genetic pool of the future will begin to select against the 'responsible' group. Outside of genetic evolution this also has a major effect on cultural evolution. There are strong correlations between fertility and low income, low education, and high religiosity. 'Responsible fertility' means children from these groups will make up an ever larger percent of all new births. Makes one question the notion of 'responsibility' in this regard.
There are also survival factors that are not so in your face as e.g. being eaten alive by a predator or starving to death. For instance obesity is becoming ubiquitous, yet it is strongly connected with a wide array of factors leading to infant death - including neonatal. Having a child means nothing if that child is not ultimately able to, in turn, also able to procreate. Another example of this issue would have been in olden times when royalty would sometimes inbreed to the point of committing genetic suicide. See, for instance, the Habsburgs.