Hardy didn't "work" for a living, to him mathematics was an obsessive hobby. He had a very strict routine; get up, walk, maths, watch cricket, tea, maths, eat, etc. He almost had the same daily routine as Kant, another bore, except the cricket.
Also, class has nothing to do with it. He was born into the imperial crown, and given his education, he was guaranteed a post in India or Africa and a "handsome life". The man was no Russel; he was an elitist who was indifferent to reaping the rewards of colonialism.
Knuth is very much a proletariat. No comparison.