Typical banana dose: 0.1 μSv per banana [1].
Typical concrete building dose: 100μSv per year [2]
ie. 1 year in a concrete building = eating 3 bananas a day.
I'm not claiming that fly ash laden concrete is going to kill anyone, but I think it's interesting that (given my assumptions) the concentrating effects of coal combustion are causing a decent proportion of concrete's radioactivity (even if the total is still negligible).
[1] https://en.wikipedia.org/wiki/Banana_equivalent_dose
[2] https://www.euronuclear.org/info/encyclopedia/r/radiation-ex...
That's the thing though, these low 'safe' levels of radiation do actually kill people. Every now and then somewhere, someone will be hit by a radioactive emission from Fly Ash that happens to hit one of their chromosomes and trigger cancer.
If Fly Ash increases cancer cases in a year by x% then we can be pretty sure that x% of cancer patients were victims of Fly Ash, it's just that x is actually fairly small and we just don't know which ones they are. But they still have cancer.
The average U content of the upper continental crust is between 2 and 3 ppm. Individual rocks will range lower and higher than that. If fly ash is only 4 ppm U, you have more to worry about from living on granitic rocks, or near a swamp than you do from fly ash.
More context: if you live on the Chattanooga shale it has an average U content of 30-120 ppm. The extensive Alum Shale (runs the length of Sweden) has U contents of 100-300 ppm. Granites will have low-mid 10s of ppm U and Th.
The report [1] linked in a comment above makes the comment Fly ash is commonly used as an additive to concrete building products, but the radioactivity of typical fly ash is not significantly different from that of more conventional concrete additives or other build-ing materials such as granite or red brick. One extreme calculation that assumed high proportions of fly-ash-rich concrete in a residence suggested a dose enhancement, compared to normal concrete, of 3 percent of the natural environmental radiation.