I guess he wants to stress that having a degree from Harvard is not as important as one might think. In his case, I guess it was the hardships that took him to the heights; maybe the Harvard degree would not have been any good for him.