My guess is the amount of skill required to get these materials. Miners can be paid pennies and they'll still do the work. Something similar to how garbage takeaway is of very high importance, but economically pretty minor.
I concede the point that small individual parts of the overall waste economy are prominent problems... but where's the way back to lithium.
Lithium is cheap enough that Elon could buy all of it with just his bitcoin. He's a guy with a use for lithium. Why not solve all possible scarcity concerns with the jingle of a purse. More realistishly... Unless there's something I'm missing, I can't see how a lithium price spike could affect the price of an electric car noticeably.
That doesn't scream scarcity.
From your comment that the lithium market is so small, yet always talked about as criticial - I guessed that perhaps that's because it's similarly easy to find and train lithium miners.
As for Elon and lithium, I remember recently watching some interview or talk that he gave, where he mentioned that he doesn't consider lithium a problem at all precicely because it is so abundant in the US.
So yes, to me it seems as well that there is no scarcity to fear. There might be a temporary shortage, but it can be solved rather quickly with money.