I read that too, and I got the impression being forced to work alone at the patent office was crucial for his 1905 Annus mirabilis papers. Lots of other great ideas (e.g. Mendel, Darwin) have come about from intellectual isolation.
I wonder how Graham would respond to that given he's so intertwined in Silicon Valley.