Computers aren't radio or CDs, there's no way to actually consume media on them without copies being made at some point.
Additionally, its not obvious who is infringing. Unlike say, downloading a torrent, people have legal access to youtube ordinarily, and pay for it. If you pay for offline access to music, why can't you use youtube-dl? Any infringement will only begin when your license to the content ends.
Additionally, the software just makes HTTP(s) requests and can work with any site. I actually have never used it to download anything on youtube.