> (the batteries in the EV may even make it a little worse)
I'm not sure it makes it worse. Lithium is a common, non-toxic element that is easily recycle-able. Pound-for-pound batteries are "just" mostly replacing steel and lead (and oil-based lubricating fluids) ICE engine components, which can be toxic, sometimes nearly impossible to recycle, and produced in very unclean smelting, machining, (and refining) operations.
It's easy to forget that the electromagnet motors in an EV are not just a lot fewer moving parts than an internal combustion engine, but are a lot fewer parts entirely. The emissions savings in all those ICE parts is presumably pretty big given how huge the relevant supply chains themselves are. (Ever glanced at a Bosch parts catalog? And that's just one supplier of dozens.)
I've never seen a good breakdown of the emissions costs of a contemporary internal combustion engine and exhaust system, but that's a much better comparison versus the emissions associated with batteries than a lot of the current battery emissions breakdowns assume as if batteries were in addition to a traditional engine, rather than almost a full replacement for one.