Teaching Language Models to Bet Honestly on Their Own Answers

This episode examines RewardingDoubt, a reinforcement-learning method for training large language models to express calibrated confidence rather than defaulting to near-maximal certainty regardless of correctness. The discussion covers why miscalibrated confidence is dangerous in high-stakes domains like medicine, legal consultation, and customer service, where deferring to a human reviewer only works if the model's stated confidence is trustworthy. It walks through the technical core of the approach: a proper scoring rule (specifically the logarithmic scoring rule) that mathematically rewards models for reporting their true beliefs rather than gaming their confidence scores, framed intuitively as a betting game where lying about certainty costs real "money." The conversation contrasts this RL-based approach with prior black-box (output-only) and white-box (internals-probing) calibration methods, including Kadavath et al.'s self-evaluation technique and Lin, Hilton, and Evans' supervised fine-tuning approach, arguing RL sidesteps the quality ceiling imposed by fixed ground-truth labels. It also details the paper's MDP formulation, where the model's answer is frozen before a separate PPO-trained pass generates the confidence sequence, and explains the two evaluation metrics—Expected Calibration Error and AUROC—used to judge whether the method actually works.