But also... isn't the entire point of "confidence scoring" to tell you how confident the model in use is? I mean, I kind of agree, Claude still tells me things it does in like 5 to 15 minutes will take many months (which for a human, maybe a few weeks).

I think JEV has shown the industry you don't need LLMs for everything, and the point of confidence scoring is to know pretty well. I think an LLM could mess up confident scoring, but for the most part, for very specialized tasks with small context windows, I could see it getting it mostly right. I do wonder if having "criteria" defined for how it should determine its own confidence score would work, but then at that point, why not just ask it to validate the criteria and you measure the confidence of the models answer, each specific answer would bear some of the weight into the confidence score anyway no?

I'm not a guru, just thinking aloud. I think there's right to be concerned, the other day a buddy showed me a screenshot (last week) of google suggesting if he has RAM problems to buy new RAM, because RAM is cheap. Models can be stupid.