It’s not unusual for clinicians to interpret differently based on patient history and other risk factors.
The same CT scan is going to be interpreted very differently for a patient with no history of cancer versus one who went into remission 1 year ago.
Any model or human will suffer from some degree of uncertainty. You can never say anything with 100% certainty.
If you know a patient had cancer before, the chance that they have cancer again is much higher usually than the general population. This is very valuable for diagnosis.
It's good to have a prior/bias to check more closely here and err on the side of caution.
It's good to check for the more likely explanations first when you have multiple competing hypotheses and limited capacity for testing.
Do you just tell the AI "this person had cancer?", or "this person went into remission 1 year ago?" or "this person was treated with 4 lines of chemo, partial remission for 1 line, non-response for 2nd line, strong response for cycles 1-3 for 3rd line, but then discontinuation due to tolerability, then 4th line complete remission after 6 cycles plus two courses of radiotherapy"?
Theoretically you could do this, but capturing all this and correctly interpreting it is far harder for AI than for a human who has been directing treatment the entire time.