The rights we're talking about here are not able to be compared to physical property rights because there is no mutual exclusion of access.
If somebody steals my physical property, I can't use it, therefore they've harmed me and limited my actions. For knowledge, IP, etc. they cannot limit my access or permission, so I do not need to be put up guards.
My core argument, I guess, is this: There is no reason to have copyright because it is effectively unenforcable and because it has no real-world grounding. Moreover, it props up an economic model that fails in exposure to modern distribution methods.