Feynman makes a good point when he's describing why he isn't a genius.
Someone shows him a blueprint and he just happens to know X because (I can't remember, something he did last week) and that's such a little thing, but everyone is "wow!" and then someone else asks him a question and really, Y is the obvious answer because square root Z is (something something something) and everyone says "wow".
It ends up that Feynman worked on a lot of stuff, all the time, and remembered a lot of it.
I guess Feynman wouldn't have remembered his phone number unless it had some interesting characteristic.