Wow, I have had exactly this idea and it makes so much sense I was surprised I never heard anyone else come to this conclusion yet, but finally I seem to have found one. My idea in a nutshell:

tax only non renewable or finite resources and not labour

Finite resources include - land use (be it housing, leisure or agriculture though desirable locations should have higher taxes) - resources (metals, minerals) - fossil fuels

Not to be taxed: - labour - wealth (having money has no environmental impact. It's the spending that matters) - added value (get rid of VAT)

I feel like all tax systems are based on what is easy to be taxed, but I have never heard a good argument on why e.g. income should be taxed, but environmental impact not