Great. Then we should keep an eye out for the actual thing that's down the slope. Forgoing useful technologies because a slope exists is like refusing to mine coal because the uranium nearby could some day be used to build atomic bombs.
Neither of your example minerals are particularly healthy for humans, nor can they be made so without extreme efforts.
But we can agree there's a world of difference. There isn't a massive international storage and handling agreement for coal and its waste products, and no country has ever been preemptively bombed over mining too much coal.