We've also been weeding out high testosterone men to some extent. For the last century, we've been putting a large number of high testosterone men in prison.
It plausibly could have a negative impact and the probability that you have children that themselves have children; I suspect that children of imprisoned fathers are more likely than the population average to die before having children themselves.