The RIAA's logic is that a youtube-dl user is equivalent to someone in a movie theater with a video camera. The situation they're concerned about is somebody downloading a music video that they've authorized to be on Youtube. They make money on Youtube views, but they're losing precious fractions of a cent when you download it and watch it offline.
This is obviously not ethically justified, but I have no idea if it's legally justified.