One of projects that I consult on involves sorting of gravel from inactive uranium mines. On site the difference between "safe gravel for construction" and "radioactive waste" (which was found out to be viable as precursor for nuclear fuel) is mostly insignificant, but easily detectable. In my house this difference is swamped by natural background radiation (see "Living in a brick, stone..." in the XKCD chart, the sorting site is open-air installation).
Edit: Somehow I think that the real danger of radiation comes from radioactive stuff that can be aspirated or ingested, not from "some well-localized thing that is radioactive" and thus any kind of "fear of radiation" is mostly irrational (given that one does not go near radiation sources unnecessarily). Otherwise it's poison as any other (and most radioactive isotopes with long half-life are primarily chemically poisonous with the radiation hazard being mostly insignificant).