You cannot amass this much wealth in a reasonable -- 40-50 y.os need not apply -- timeframe if you have a family commitment. Even significant others may become detrimental to that end, if they cannot accept that they're #2 now. Close relationships take time. An absolute fuckload of time.
In order to gain that amount of wealth, it takes years of 70+ hour work weeks. You will neither have the time nor inclination to spend time with tour loved ones if you're truly operating on all cylinders. If you're not, that's fine. That's your choice, but there's many more out there that don't have those kinds of personal hangups and will dedicate more time to winning.
You would not be able to understand unless you've been there.
Oh and Warren Buffett is not a paragon of family values. He has had falling outs with and disowned one of his children -- the rest is washed away by PR firms.
Gates too. Melinda would barely see him in the early days of MSFT. Neither would his first newborn child, while Gates was wrapped up in the antitrust suits.
These people were businessman first and foremost. They did not get to the top of the food chain by spending quality time with their children. Theire businesses (or passions) will always be #1 to them.
Invariably, these men have spent less time with their children and wives than any poor south americans -- who happen to have higher emotional wellness than their American counterparts.
This isn't about not having a family. Many high perfomance men have families, but they always take a back seat to their missions. Perhaps in later life when they're more established, that changes; I've seen it happen.