They were too focused on the technicalities of the problem too see the big picture, n issue I see a lot in data science students who often struggle to grapple with the problem they're supposed to be solving.
They were too focused on the technicalities of the problem too see the big picture, n issue I see a lot in data science students who often struggle to grapple with the problem they're supposed to be solving.
I got into calibration through text retrieval where the TREC methodology rates relevance functions by rank in such a way that you don't get points for being calibrated or well-calibrated.
There are numerous things mainstream text retrieval systems don't do, most notably it is hard to build an alerting feature without calibration. Practically you have to set some relevance threshold to avoid getting too much irrelevant stuff.
I calibrated a rather good search engine by fitting a curve to the score and found that the best it would ever give is p=0.7 to be relevant and that it gave that very rarely. So even with a calibrated score you wouldn't be able to set a very high threshold or expect to get many documents. Watson gets at this problem by having a large number of question answering models each which has a high p to be right when it is right but each of which also has very low recall, something you can do when using p as a universal score.
Sounds like the rest of the class didn't read the fine print if that was the model that won.