Your answers to this are going to affect how much money is "FU money" for you.
If it means not having to work, you might also consider why you need to live in Silicon Valley. It's not a particularly great place to live other than the job opportunities, and a million dollar house there costs a fraction of that in most of the rest of the country.
Also, can I ask if you experience is first hand? If not, be careful with appearances. I just recently discovered that someone I've known for years has significantly more money than their lifestyle would indicate. Not everyone lives extravagantly just because they have money.
My experience with $5M is in fact firsthand, and it feels a lot closer to "middle class" than to "rich". Maybe it's because I live a non-extravagant lifestyle and I just need to buy myself a Ferrari :-) But seriously, what I'd like is to find out from people who actually know, if there's a financial level at which life changes qualitatively. E.g. is there a level at which it makes sense to stop working and spend your time managing your charitable foundation and hanging out at TED and Davos?
Not that I know from personal experience, but my best guess on that level of wealth would be 100s of millions. That's a different level of existence.
I know plenty of people with millions, they seem more or less ordinary, though their leisure activities exceed the norm. The couple of people in the 10s of millions range that I'm somewhat acquainted with have very nice living environments (think estates), but don't have foundations. The one person I know with foundation/ted/davos is in a family with the 9 figure range.
Don't really know silicon valley folks, though.
At least in the valley, most people still live below their means. I find that on the East Coast and Southern CA that those with expensive lifestyles are usually heavily burdened by debt. There are many many wealthy people in the valley who live in simple apartments and walk everywhere, not needing the flashy car or flashy clothes to justify their existence and place in society.
Having said that, there are an extraordinary number of genuinely wealthy people in Socal. It's odd at times.
I work with a lot of doctors that make $200 - $300k a year. And, I'd say that makes the comfortably upper middle class in the Bay Area. Couples where both are nurses can make $250 a year without working very hard.
Salaries at those levels make it very feasible to buy a house and take a nice vacation or two to Hawaii, Thailand or Europe. They drive nice cars and have lovely electronics to play with.