This is not surprising, since the model is just given one piece of information and it's the piece of information the CNN has been trained to make its whole decision on.

I'm not a dermatologist, but I imagine they don't make their decisions on one image alone (and the decision probably isn't "is this cancer" but "should we take a biopsy"). The LLM can take into account additional information such as the patient's age, occupational exposure and medical history, the CNN can not.

But then again, maybe it's useful to have a tool which you know does not use any information besides the image to make the call.