But since this is really a "should" question (aka about values), theres not really a right answer. We can mostly just talk about fairness, of which there are many different conceptions.
My take: Society has to work hard to preserve the billionaire's wealth. You can't be king without an army, and billionaires exist because of laws, police and a social fabric that lets them keep owning so much stuff. You would otherwise have to be a warlord to keep your hands on so much, and that has its own costs and risks.
Perhaps we could look at it as a fee or compensation for the aggregate social guard labour required to preserve the social order that ensures the safety of the billionaire's assets.