I think there is a bit stronger version of your argument:
The wealth that I have is intrinsically tied to me personally: It's a business that I'm running, an idea I'm pursuing, etc. Taking it away from me destroys it.
The discrepancy comes from confusing money with wealth. You can take my $100 and then you will have $100. But if you take my business away from me, you will end up with very little. And so will everybody else that was benefitting from the business that I was running.