The problem is that there's a certain segment of the population that views that right so broadly that it encompasses a bunch of stuff I and most others don't agree with.

Property rights are weird. We all mostly agree that no one should be able to steal your phone off your desk at work. We very much don't agree that governments should not be able to tax income. When someone says "controlling your own wealth should be a basic human right", I don't know whether I agree or not until I know quite a lot more about what that person means. Which probably means I disagree that it's a basic human right.