The problem here is confounding. Do we think there are no causes of fatherlessness that are also causes of poorer life outcomes (hint: there are plenty). Given this it's very hard to see clearly how much impact this makes, all the personal anecdotes here notwithstanding.