Can model confidence signals replace explicit external reward functions?
This explores whether a model's own confidence — and other signals it computes internally — can do the job we normally hand to an external reward model or verifier, and where that substitution holds up versus where it breaks.
This explores whether a model's own confidence — and other internally-generated signals — can replace the external reward functions and verifiers that reinforcement learning usually depends on. The short version the corpus offers: yes, increasingly, and confidence is just one member of a growing family of self-supplied signals. The most direct evidence is RLSF, which ranks a model's reasoning traces by the confidence it assigns to its own answer span, turning that into synthetic preferences that sharpen step-by-step reasoning — and, notably, *repair* the calibration damage that human-feedback training tends to cause Can model confidence work as a reward signal for reasoning?. So confidence isn't just a cheap stand-in; it can fix a problem that explicit rewards create.
What's striking is that confidence is part of a broader convergence. A 2025 survey of 'verifier-free' RL finds the field independently landing on three substitutable patterns: the model judging its own outputs pairwise replaces the reward model, the model's internal belief-shift replaces the critic, and rich self-distillation replaces the explicit reward signal entirely Can language models replace reward models with internal signals?. You can see each pattern fleshed out elsewhere: belief-shift RL uses the log-ratio of how the model's probability estimates move toward a solution as a dense, per-turn reward — no critic network needed — and lets small models beat larger baselines Can an agent's own beliefs guide credit assignment without critics?. Test-Time RL goes further and manufactures reward from majority vote across repeated samples, training on unlabeled data because consensus tends to track correctness Can models improve themselves using only majority voting?. Post-Completion Learning even teaches the model to compute its own reward in the unused sequence space after its answer, internalizing evaluation at zero inference cost Can models learn to evaluate their own work during training?.
But here's the catch the corpus insists on: the *shape* of the reward matters more than whether it's internal or external. Binary correctness rewards — the simplest external signal — actively reward confident wrong answers, because nothing penalizes being sure and incorrect; bolting on a Brier-score term mathematically restores calibration Does binary reward training hurt model calibration?. Ternary rewards that make abstention a learnable option cut hallucinations sharply Can three-way rewards fix the accuracy versus abstention problem?. And a scalar reward — internal or not — throws away information: agent feedback decomposes into 'how well did this go' (evaluative) and 'how should it change' (directive), and confidence-style signals capture the first while discarding the second Can scalar rewards capture all the information in agent feedback?. There's even a counterintuitive result that negative reinforcement alone — just suppressing wrong trajectories — can match full RL while preserving diversity Does negative reinforcement alone outperform full reinforcement learning?.
The deeper limit is what any reward signal can do at all. Several notes converge on the finding that RLVR — the explicit-verifier paradigm these methods aim to replace — doesn't actually expand what a model can reason about; it just narrows sampling toward solutions already latent in the base model, which is why spurious rewards work nearly as well as correct ones for a well-pretrained model Does RLVR actually expand what models can reason about? What does reward learning actually do to model reasoning?. If reward is mostly *activating* existing capability rather than teaching, then confidence signals are a plausible substitute precisely because the bar is lower than it looks — and that's the thing you might not have known you wanted to know. The fact that RL touches only 5–30% of parameters in stable, near-identical subnetworks regardless of algorithm reinforces this: these methods are steering, not rebuilding Does reinforcement learning update only a small fraction of parameters?. So confidence can replace the reward function — but only because the reward function was doing less than we assumed.
Sources 12 notes
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Late-2025 RL literature independently converges on three patterns that replace different RLHF components: pairwise self-judgment replaces the reward model, internal belief-shift replaces the critic, and rich-feedback self-distillation replaces explicit reward signals. Each emerges from the policy's own computations, making the trained reward classifier optional.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
Test-Time RL generates reward signals by majority voting across repeated samples, enabling policy improvement without ground-truth labels or trained reward models. This approach works surprisingly well because consensus answers tend to be correct, creating a bootstrapping loop where test-time compute enables training that improves the model.
Post-Completion Learning exploits unused sequence space after model output to train self-assessment capabilities during training while maintaining zero inference cost. The model learns to compute its own reward functions, internalizing evaluation rather than relying on external reward models.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
TruthRL uses three distinct rewards (correct +1, hallucination -1, abstention intermediate) to make abstention learnable. Across four benchmarks, this reduced hallucinations by 28.9% and improved truthfulness by 21.1% compared to binary reward RL.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.