If we spent the same $/TWh on coal/oil safety as we do on nuclear, maybe this would look different. Obviously, coal and oil are simply used, abundant, and cheap.
There are also a considerable number of deaths from mining accidents, which of course are drastically smaller for nuclear than for coal. Considering the relative amounts of mining required, this should not be a surprise.