Does negative reinforcement alone outperform full reinforcement learning?
Can training with only penalty signals for wrong answers match or exceed full RL approaches? This challenges the conventional assumption that reward design requires both positive and negative signals.
Decomposing RL's learning signal into positive sample reinforcement (PSR) and negative sample reinforcement (NSR) reveals a surprising asymmetry. Training with only negative samples — penalizing incorrect responses without ever reinforcing correct ones — consistently improves performance over the base model across the entire Pass@k spectrum (k up to 256), often matching or surpassing full PPO and GRPO.
The mechanism is straightforward through gradient analysis: NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs. It refines existing knowledge rather than introducing entirely new behaviors. This is because penalizing a wrong answer doesn't point toward any specific correct answer — it lets the model's own prior determine where the freed probability mass flows.
Positive-only reinforcement creates the opposite problem. It improves Pass@1 (the model gets better at its top-ranked answer) but degrades performance at higher k because it concentrates probability mass on rewarded trajectories, reducing diversity. Since Does policy entropy collapse limit reasoning performance in RL?, positive reinforcement actively contributes to the problem while negative reinforcement sidesteps it.
This reframes how we think about RL for reasoning. The conventional framing is that RL rewards correct behavior. But the evidence suggests that penalizing incorrect behavior may contribute more to performance than reinforcing correct behavior — especially when diversity matters. The model already contains good solutions in its prior; it just needs help avoiding the bad ones.
The practical implication is that reward design for reasoning RL may be over-engineered. If suppression alone gets you most of the way, the elaborate reward shaping and process supervision architectures may be solving a problem that's already largely solved by the base model's prior distribution.
Inquiring lines that read this note 75
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do self-generated feedback mechanisms enable effective model learning?- Can unified policies handle negative feedback and critique transformation simultaneously?
- What makes Effective Rank Acceleration a stable training signal for dual-channel incentives?
- What makes content informative and not-yet-mastered for reinforcement during pretraining?
- How do reward models and self-improvement mechanisms interact in training?
- Why does binary reward forcing degrade model calibration?
- Can log-likelihood loss combined with binary rewards achieve calibration?
- Can model confidence signals replace explicit external reward functions?
- Can checklist-based rewards fix judgment problems in RL training?
- What behavioral changes occur during reward learning training?
- How does reinforcement learning compare to differentiable joint training for RAG?
- Can negative reinforcement alone match full RL performance on domain tasks?
- Does negative reinforcement alone achieve what full RL training accomplishes?
- Why do spurious rewards work nearly as well as correct ones?
- When does outcome reward signal become informative during model training?
- How do reward signals in RLVR interact with pretraining biases?
- How does 93% reward reliability compare to other RL noise sources?
- Are different reward signal sources substitutable in verifier-free RL?
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- What makes reward signal sources substitutable across verifier-free RL patterns?
- What distinguishes high-signal prompts from low-signal ones in RL training?
- Why do spurious reward signals improve reasoning for some pretrained models?
- Do outcome-only reward signals miss step-level errors that compound later?
- Why do different models respond differently to spurious rewards?
- Can binary judge feedback replace external reward signals for skill learning?
- Can the same variance signal work as both reward and query filter?
- What makes binary rewards more effective than richer reward signals?
- When does a task lack a meaningful multi-dimensional reward structure?
- How do internal model mechanisms escape token-level reinforcement signals?
- Can importance sampling reduce variance in off-policy reward estimation?
- Can negative feedback through critiques achieve the same steering flexibility as positive preferences?
- How do relational reward signals compare to absolute preference encodings in RL?
- Can we reverse the instruction-following deficit through targeted training?
- Can light human signals steer already-learned behavior without preference labels?
- What information do next-state signals contain beyond what scalar rewards capture?
- How do evaluative versus directive signals differ in next-state training?
- What deployment modes work best for trajectory-aware reward signals?
- Can early experience replace external rewards as a learning signal?
- Why do outcome-only rewards fail to optimize long-horizon agent behavior?
- How does forced exploration through diversity rewards differ from suppression-based negative reinforcement?
- Why does positive reinforcement degrade diversity at higher k values?
- How does active selection of training content differ from random reinforcement sampling?
- Can meta-reinforcement learning explain why this bias pattern emerges rationally?
- What makes pretraining composition more important than reward engineering?
- What makes process-level supervision better than outcome-only reward signals?
- How do process-level rewards compare to environment-extracted next-state signals?
- How does process-based reward differ from outcome-only reward in training?
- How does negative reinforcement redistribute probability without guiding toward correct answers?
- Is elaborate reward shaping necessary if the pretrained prior already contains good solutions?
- How do reward model biases cascade into downstream optimization failures?
- Why do spurious rewards work for some models but not others?
- Can log-probability ratios resist reward hacking better than learned PRM signals?
- What happens when variance in reward signals comes from a noisy model?
- How do you extract reward signals when all rollouts fail?
- Can structured rewards still teach models when spurious rewards also work?
- What makes current learned reward models fail across different domains?
- What happens when error accumulation and preference signal collapse occur together?
- What makes preventative lessons from failures more valuable than success patterns?
- What makes abstention a learnable behavior instead of a default penalty?
- Can abstention behavior transfer from small models to frontier models?
- Do negative constraints require fundamentally different training signals than positive instructions?
- Can a rejected-edit buffer work like hard negatives in contrastive learning?
- Why do structure-targeted training negatives fail to fix the underlying problem?
- Why does negative experience transfer better than positive examples alone?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does RL improve domain reasoning by adding knowledge or removing it?
When reinforcement learning improves reasoning in specialized domains like medicine, is it teaching models new facts or preventing them from using wrong ones? Understanding this distinction matters for how we design RL training.
directly supports: pruning IS negative reinforcement at the reasoning path level
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
extends: positive reinforcement actively causes entropy collapse; negative reinforcement avoids it
-
Does reinforcement learning update only a small fraction of parameters?
Investigating whether RL algorithms consistently modify only 5–30% of model parameters across different LLMs and RL methods, and what structural properties those sparse updates possess.
complementary: if RL only touches 5-30% of parameters, negative reinforcement may be the primary mechanism for this sparse selection
-
Does the choice of RL algorithm actually matter for reasoning?
Expert Iteration, PPO, and RC-RL show similar performance on reasoning tasks. The question is whether algorithm choice drives results or whether something deeper—like the pretrained model itself—sets the real limits.
supports: if negative reinforcement alone suffices, algorithm choice matters even less
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
- Information-Theoretic Reward Decomposition for Generalizable RLHF
- Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
- A Survey of Reinforcement Learning from Human Feedback
- Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Efficient Reinforcement Learning via Large Language Model-based Search
- Reinforcement Learning with Rubric Anchors
Original note title
negative reinforcement alone matches or exceeds full rl by suppressing incorrect trajectories and redistributing probability mass