You bet your ass it does. I wouldn't want to live within a hundred miles, but at least we only really need one such dig site for the planet. Economics being what it is, there are more, but not a lot.
Now go look at a coal mine, and multiply the damage that does by the number of mines.
We can't eliminate all damage. We can minimize it, for which nuclear power is helpful.
In larger picture these side streams don't matter much. What happens in Kazakhstan's (36% of all U production), Canada's and Australia's uranium mines does. They account for 64% of global uranium production. Australia has 23% of the world's uranium deposits, CAN 12%, KAZ 10%.
About 85% of current needs are met with digging, 15% comes from secondary sources.
It's interesting that two of the biggest producers of uranium do not utilize nuclear energy themselves. Even Canada produces only 12% of their electricity with nukes (France 78%). Good business even if you don't personally believe in it?