That implication is not a logical necessity. There is a big distinction: if I take away the thing you made with your hands from you, you have one fewer thing, and I have one more thing.
If I learn or copy what you have in your mind, both of us have something. I cannot take it away from you.
Also, the whole phrase "Intellectual Property" is a misleading thing that should not be used in any reasonable discussion, because it derails the arguments by unifying a bunch of pretty different things[1].
We should be calling copyright, patents, and trademarks what they are: they are not properties, they are monopolies authorized by the government (trade secrets may be a bit distinct here). Perhaps authorized monopoly is a more accurate term than "intellectual property". cf. [2]
[1] https://www.gnu.org/philosophy/not-ipr.en.html
[2] http://www.locusmag.com/Perspectives/2016/11/cory-doctorow-s...