Why not just use conda for envs and deps (or env-specific pip), and install youtube-dl etc. via your platform's package manager?
In principle, it's a good idea; in practice, I'm not satisfied. On Windows, it's an easy solution, especially for packages that depend on non-python dependencies (e.g. hdf5).