Thanks kscaldef and detst for your replies. I do think a lot about moving out of Silicon Valley especially since the state of California is in such a financial mess. Theoretically, I could quit my job now, but it wouldn't make much financial sense, and I enjoy where I'm working.
My experience with $5M is in fact firsthand, and it feels a lot closer to "middle class" than to "rich". Maybe it's because I live a non-extravagant lifestyle and I just need to buy myself a Ferrari :-) But seriously, what I'd like is to find out from people who actually know, if there's a financial level at which life changes qualitatively. E.g. is there a level at which it makes sense to stop working and spend your time managing your charitable foundation and hanging out at TED and Davos?