Carbon is not the only possible cost to the environment. Building batteries is not exactly easy on the planet either.
You are right about Cobalt, but it is already being reduced/removed.
It's not even comparable to fossil fuel mining (even though technically it is fossil fuel mining). Because it's recyclable, so we aren't "losing" any material, in the form of converting it to an unproductive/hazardous byproduct.