Coal ash, tailings, hydrocarbon extraction byproducts; those are also dangerous. Look at cancer rates near refinery areas in Louisiana or Houston -- you might imagine a nuclear bomb went off in those areas based on the cancer rates.
Nuclear waste has the unique property that it is very easily detected. This is actually a good thing! It means we can isolate it, test for isolation, and stay away. The other upside is that there is less of it. Contrast that with something like the Kingston fly ash spill. There is not enough U-235 on earth to spill that much: https://en.wikipedia.org/wiki/Kingston_Fossil_Plant_coal_fly.... That is a single event (with a death toll similar to Chernobyl!) The picture for semiconductor manufacturing or battery manufacturing is not perfect either.
Short sighted? Removing a zero carbon clean energy source from the table when decarbonization is urgently needed -- that is short sighted.