So many anecdotes that he related about all the prominent scientists involved during the Manhattan Project.
- liked to read books whilst driving
- on his death-bed, reciting, by heart and word-for-word, the first few lines of each page of Goethe's Faust
- always had a stack of books next to his bed that he was reading
It seems plausible that if he could retain everything he read he would want to read as much as possible.
I haven't come across any first-hand account of people explicitly saying this was bullshit, other than skepticism from people who never knew him. There's always going to be skepticism on outlier events.
Here's an interesting post about this memory: http://overweeninggeneralist.blogspot.co.za/2011/11/john-von...