Do you own a personal jet or a personal yacht? Bill Gates does (or definitely can).
Who remembers Tom Vu? And his yacht? http://youtu.be/c5OOHotxAYk
Never being able to go anywhere spontaneous without your security guards. Everyone looking to take your money. Having to watch your words in everything you say because you are being filmed by strangers 24/7. Can't walk into a coffee shop without being recognized and hassled for a selfie cause some vain dude/girl wants to show off to his friends and get social network "likes". Etc.
The only way to get away from all those problems is probably to buy a private island, build a big house, and trap yourself inside. Which a lot of wealthy people kind of do in one way or another.