Oil, coal, gas, hydro, etc inflict casualties gradually and predictably. They are, generally speaking, not capable of the sudden region-wide catastrophic disasters that nuclear power is capable of. Those are rare events. The Poisson distribution, with regard to inflicting mass casualties, is completely different for nuclear power.
We haven't had a true nuclear catastrophe yet. We came close with Chernobyl and Three Mile Island, but those events were not nearly as bad as they could have been. The number of casualties could have been orders of magnitude higher.
In other words, our sample size with nuclear power is small. All it would take is one nuclear catastrophe and the safety stats averaged over the last century suddenly look much different.