Yes, downloading is akin to making a copy without consent of the copyright owner. (Edit: At least according to the Berne Convention, which most countries have signed.)
Edit2: I think dchest is right and illegality of the act of downloading does probably not follow from the Berne Convention, sorry. Distribution (offering for download) certainly is and I think at least according to German law the resulting copy has to be destroyed. Mind you, I'm no lawyer and might be mistaken.