They don't need to grow, efficiency can grow though as very aptly demonstrated in the 20th century.
Thermodynamics sometimes say that you can't. Also, if I can mine $x coins for y energy, I will generate a bit more of $money and remove natural resources. Value of coins may not keep their value in 2000 years, but co2 ppm/ecosystems/metals/whatever won't be there anymore, unless we know how to deal with them, that we may or we may not.