I disagree with their point of view because:
1. My neighbour having enough food to eat and a warm house is a form of wealth FOR ME.
2. If wealth that is generated is taxed, this means that it is harder to accumulate wealth, this means that the people that do manage to accumulate wealth have better qualities than the people that do so in a tax free environment. And thus they make better decisions about how to allocate their money.
3. Billionaires are not accountable to anyone for how they spend their wealth. This is fine if they are all like Musk. But they are not. Of the countless number, only two are trying to build rockets to Mars. Counter examples are Osama Bin Laden and the guys who funded the NRA. If the wealth is taxed then it is fought over in a shared space in which we all have a say, no matter how small.
I like the argument that Rawls made in "A theory of Justice", where he said that when you are deciding how to make a fair society you should think from a perspective in which you do not know what role you will have in that society.