If you have a few millions in an index fund and are discrete about it, no one is going to take it.
There are many rich people who live ordinary lives.
There are many rich people who live ordinary lives.
You should also not be investing more than 25% of your assets into a single asset class, so you need to have $6-12MM of assets (range depending on considerations) in the first place.
I think having even $3MM is a reasonably called 'rich', but others may disagree.
Also, if you’re going to posit funds in other asset classes, you should also include the worst-case returns of those funds in the analysis.