One is that people often mix political and legal matters in colloquial discussions of such topics. I frequently catch myself doing it too.
It is clearly not in the interest of most people for functionality that youtube-dl offers to be illegal. In fact, most of the time it isn't really illegal, in a de facto sense, since it is never acted upon and prosecuted.
However, sometimes a shady organization such as the RIAA acts upon this legal grey area and tries to push the needle in its own direction. When people then say "There is actually nothing wrong with this, this is definitely not illegal.", they are actually making a political statement without realizing. They are really saying: this shouldn't be illegal because it makes no sense that it is so. Since people actually hold all the power, this act makes this the reality in a way, but in order for this to completely materialize, the will of the people must first be focused so that it is reified into law.
The other problem is that the world is not the US and there are many jurisdictions where circumventing copyright protections and/or downloading a media file for personal use are not problematic at all. This fact also often seeps into the discussion, but sometimes only implicitly, so people often talk past each other.