Can an agent's own beliefs guide credit assignment without critics?
Explore whether an agent's shifting probability estimates toward the correct answer could serve as a self-contained reward signal for long-horizon reinforcement learning, eliminating the need for separate process reward models or external verifiers.
Long-horizon RL suffers from sparse trajectory-level rewards. The standard fixes — process reward models trained on step-level annotations, external verifiers, LLM-as-judge — all require additional supervision infrastructure. PRMs need expensive step-level labels. Verifiers exist only for verifiable domains (math, code). Judges introduce their own reward-modeling biases.
ΔBelief-RL (2602.12342) finds the credit signal inside the agent itself. At each interaction step, compute the agent's current probability assigned to the target solution. Compare it to the probability before the interaction. The log-ratio of sequential beliefs is the ΔBelief reward — a dense, turn-level signal that reinforces actions which shift the agent's internal world view toward the correct solution. Actions that increase belief in the target get rewarded; actions that don't, don't.
The elegance is that no separate model is needed. The agent's own log-probabilities on the correct outcome are the value signal. There is no critic to train, no PRM to maintain, no judge to query. The relatively inexpensive step is measuring log-probabilities on the target — a single forward pass per turn.
Two properties make this work. First, it is general-purpose: applies to any task where the correct final outcome is available during training (which is most supervised settings). Second, it is noise-robust to over-optimization: PRMs can be exploited because their reward signal is a learned approximation; ΔBelief is grounded in the model's own evolving probability assignment, which is harder to game because the only way to increase log-probability of the target is to actually integrate information that supports it.
Empirically, ΔBelief-RL on 20Qs trains CIA models at 1.7B-4B scale that outperform prior SOTA multi-turn methods and even 670B models. Performance generalizes to extended interaction horizons beyond training and to OOD applications (customer service, personalization).
The mechanism aligns with Can conversations themselves personalize without user profiles?: both reward uncertainty reduction. But ΔBelief's signal is about the target's probability specifically, while curiosity reward is about general uncertainty over user type. ΔBelief is information-theoretically tighter — it rewards moves toward the actual answer, not all moves that increase clarity.
The broader implication: in any setting where the model has ground-truth final outcome, the model's own probability shift can serve as dense intrinsic reward. The reward model is not load-bearing.
Inquiring lines that read this note 80
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What prevents or enables error accumulation in self-training systems?- Does self-conditioning improve belief-behavior alignment better than external priors?
- Why does self-judgment of success or failure work without ground truth labels?
- Does the generation-verification gap define where self-rewarding actually works?
- Can systems recognize and abstain on judgments rather than hallucinating preferences?
- Can subjective tasks be delegated without human feedback loops?
- Why do weak belief tracking and conservative actions trap agents in low-information states?
- Why do agents fail to internalize value from informative observations?
- Can agents escape weak belief tracking and conservative action selection traps?
- How does poor belief tracking cause agents to keep acting past the point of usefulness?
- Can agents reliably estimate the stakes or consequences of an action?
- How do agents ground their judgments in evidence instead of pattern matching?
- When is stochastic agent behavior actually desirable rather than problematic?
- How do we assign confidence and polarity scores to belief edges?
- Can reward model training be automated without changing feedback mechanisms?
- How do probability-based rewards compare to self-consistency as training signals for reasoning?
- How does self-consistency compare to confidence as a proxy reward signal?
- Can intrinsic reward signals extend beyond mathematics to medicine and law?
- What distinguishes verifiable rewards from preference-based rewards in unified training?
- Can model confidence signals replace explicit external reward functions?
- How does belief-shift reward compare to curiosity-driven and process reward approaches?
- Can log-probability ratios resist reward hacking better than learned PRM signals?
- Why does belief-shift reward enable smaller models to match larger baselines?
- Can an agent's internal probabilities serve as value signals across domains?
- Does belief-shift credit assignment generalize to tasks without ground-truth outcomes?
- Can binary judge feedback replace external reward signals for skill learning?
- What happens when variance in reward signals comes from a noisy model?
- Can verifiable rewards during pretraining replace costly human preference labeling?
- Are different reward signal sources substitutable in verifier-free RL?
- Can structured rewards still teach models when spurious rewards also work?
- How does belief-shift credit assignment compare to process reward models?
- How do pairwise self-judgment and internal belief-shift replace verification differently?
- What makes reward signal sources substitutable across verifier-free RL patterns?
- Can light human signals steer already-learned behavior without preference labels?
- What mechanisms do peer predictions use to generate reward signals for training?
- What information do next-state signals contain beyond what scalar rewards capture?
- How does credit assignment across objectives differ from credit assignment across time?
- How do agent actions change state that reward procedures later read?
- What within-run behavioral dimensions reveal where long-horizon agents succeed or fail?
- Do outcome-only reward signals miss step-level errors that compound later?
- What makes process-level supervision better than outcome-only reward signals?
- How does reward model training permit spurious correlations in scoring?
- Is elaborate reward shaping necessary if the pretrained prior already contains good solutions?
- How do process-level rewards compare to environment-extracted next-state signals?
- Can self-supervised methods replace human annotations for process reward models?
- What information-theoretic framework explains why process rewards beat outcome only?
- How does credit assignment work across many sequential decision steps in language models?
- Why does self-segmentation into chunks-of-thought matter for reward models?
- Why do outcome-only rewards fail to optimize long-horizon agent behavior?
- Can agents revise their beliefs predictably when presented with interventions?
- Can early experience replace external rewards as a learning signal?
- How does effective feedback retention govern long-horizon agent reliability?
- How do agents decide when to abstain from contributing?
- What does it mean to infer user goals probabilistically under uncertainty?
- Does common ground alignment require explicit rewards to emerge?
- How do reward models as policy discriminators differ from labeled preferences?
- What makes reward models fundamentally different from policy discriminators?
- Does pairwise self-judgment avoid reward model scaling problems?
- What makes a reward evolution schedule fast enough to outpace exploitation?
- Can simple intrinsic reward signals emerge as effective drivers of complex capability in agents?
- What deployment modes work best for trajectory-aware reward signals?
- When does outcome reward signal become informative during model training?
- What makes a sub-goal verifiable enough to provide dense feedback signals?
- What triggers control processes to act on stored preference knowledge?
- Can models detect when their own trajectory is on-policy versus off-policy?
- How can verifier-free reinforcement learning handle reasoning without task-specific checks?
- What hard-to-verify tasks will remain resistant to reinforcement learning?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can conversations themselves personalize without user profiles?
Can a conversational AI learn about user traits and adapt in real time by rewarding itself for asking insightful questions, rather than relying on pre-collected profiles or historical data?
both reward uncertainty reduction; ΔBelief is target-specific, curiosity reward is type-general — different information-theoretic targets
-
Can we reward reasoning steps without human annotation?
Existing RL for reasoning uses only final-answer rewards, causing models to produce wastefully long chains. Can information theory provide dense, automatic feedback for individual reasoning steps?
L2T uses PAC-Bayes/Fisher; ΔBelief uses log-ratio of sequential beliefs; both convert outcome correctness into dense step-level reward without annotation
-
Can environment feedback replace scalar rewards in policy learning?
Can rich tokenized feedback from environments serve as a direct learning signal for policies, without relying on compressed scalar rewards? This matters because scalar rewards discard information needed for credit assignment.
convergent verifier-free move via different mechanism: SDPO uses feedback-conditioned self-teacher; ΔBelief uses belief-shift on target
-
Can reward models learn by comparing policies instead of judging them?
What if reward models worked as policy discriminators—measuring distance to a target rather than encoding absolute preferences? Could this eliminate the need for manual preference labels and scale across domains?
three independent paths to RL without external preference labels are converging
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Intrinsic Credit Assignment for Long Horizon Interaction
- Learning to Reason without External Rewards
- Can Large Reasoning Models Self-Train?
- Reinforcement Learning via Self-Distillation
- Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
- Reward Reasoning Model
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- RLPR: Extrapolating RLVR to General Domains without Verifiers
Original note title
belief-shift toward the target solution is a dense intrinsic reward — log-ratio of sequential beliefs provides per-turn credit without separate critic or PRM