Exploring Confidence as a Reward to Advance LLMs Reasoning
Published in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2026
We propose Confidence-as-a-Reward (CRew), a simple, training-free approach that uses a model’s token-level confidence in the final answer as a reward proxy, particularly for close-ended problems. On MATH500 and RewardMATH, CRew achieves the best performance among training-free reward models and even surpasses most trained reward models. We further introduce CRew-DPO, which constructs preference data from confidence scores and correctness signals; finetuning with it consistently outperforms existing self-training methods.
Also released on arXiv as “Confidence as a Reward: Transforming LLMs into Reward Models”.