Others have noted Feynman, Ramanujan. Walter Pitts is another one, who had a very rough childhood with poor, working class parents.