That would actually be a sound argument: If we assume a linear damage model for radiation exposure (which is a generally accepted model for these circumstances), then there is no safe dose. Instead, any exposure increases your lifetime risk of cancer. It is therefore not useful to talk about how medical imaging causes an individual case of cancer.

But for a larger population we can figure out what the cancer risk without medical radiation exposure would be, and what the observed probability is. We can then ascribe the difference to these medical tests. Obviously there are a number of ways to fuck up this analysis. Geological differences in the natural radiation dose shouldn't be underestimated. However, almost half of the lifetime radiation dose of a human does come from medical applications, and it would certainly be good to reduce that.