I think there was such a selection pressure when we were nomadic hunter-gatherers. Agriculture is a very recent development anthropologically, so we were still in the mode of one-kid-per-pregnancy, though multiple might have been advantageous. It's possible we developed higher fertility due to that, but we didn't have enough time for that new pressure to kick in.