But the switching power supply that regulates the mains voltage to 5v (or whatever is required) is always on, and although they are extremely efficient the loses do add up.
Arguably it should be possible for the phone to switch off the supply before the switching power supply, but this will add complexity and cost to the product. And because the power supply is normally seperate from the phone some kind of signalling from phone to power supply would need to be implemented. Again adding complexity and cost.
As mentioned in the article, http://www.withouthotair.com/ is well worth the read, and will help you understand what is actually worth doing to lower energy use and carbon footprint. (The book is considerably better than the appearance of the website would suggest!)