Piles of wealth require protection. Without government protection, they would be expropriated without considerable expenses on private security.
Income tax is the tax that is hard to justify. Wealth taxes are taxes to protect wealth, and sales/transaction taxes are taxes to enforce sales and transaction agreements.
Libertarians believe those should be the only functions of government. If you don't even believe in those, you're an anarchist, or maybe even a Mad Maxist.
edit: imagine the absurdity of people sharing a rented shed paying as much for fire and police protection as a person in a mansion.