The difference is that you can optimize the current banking system for lower power consumption. Bitcoin mining will always be a race on who can waste the most resources/electricity for mining.
In essence, Bitcoin's harm to the ecosystem is almost directly proportional to Bitcoin price.
Just imagine everyone using this more efficient hardware to save money on electricity, not increasing the mining difficulty. Eventually you'll have the risk of a single entity (e.g. a government organization) just buying enough hardware to have more than 50% of the hashing power, taking over the whole blockchain.
The second argument is something that is independent from the energy problem. The danger always exists, that some entity is taking over the blockchain.