The RLHF process performed on that chat models actively confuses their ability to estimate confidence. The raw models are quite well calibrated though