Does negative reinforcement alone outperform full reinforcement learning?
Can training with only penalty signals for wrong answers match or exceed full RL approaches? This challenges the conventional assumption that reward design requires both positive and negative signals.
Decomposing RL's learning signal into positive sample reinforcement (PSR) and negative sample reinforcement (NSR) reveals a surprising asymmetry. Training with only negative samples — penalizing incorrect responses without ever reinforcing correct ones — consistently improves performance over the base model across the entire Pass@k spectrum (k up to 256), often matching or surpassing full PPO and GRPO.
The mechanism is straightforward through gradient analysis: NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs. It refines existing knowledge rather than introducing entirely new behaviors. This is because penalizing a wrong answer doesn't point toward any specific correct answer — it lets the model's own prior determine where the freed probability mass flows.
Positive-only reinforcement creates the opposite problem. It improves Pass@1 (the model gets better at its top-ranked answer) but degrades performance at higher k because it concentrates probability mass on rewarded trajectories, reducing diversity. Since Does policy entropy collapse limit reasoning performance in RL?, positive reinforcement actively contributes to the problem while negative reinforcement sidesteps it.
This reframes how we think about RL for reasoning. The conventional framing is that RL rewards correct behavior. But the evidence suggests that penalizing incorrect behavior may contribute more to performance than reinforcing correct behavior — especially when diversity matters. The model already contains good solutions in its prior; it just needs help avoiding the bad ones.
The practical implication is that reward design for reasoning RL may be over-engineered. If suppression alone gets you most of the way, the elaborate reward shaping and process supervision architectures may be solving a problem that's already largely solved by the base model's prior distribution.
Inquiring lines that read this note 87
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do different post-training structures affect learning outcomes?- Can unified policies handle negative feedback and critique transformation simultaneously?
- How do evaluative versus directive signals differ in next-state training?
- Does negative reinforcement alone achieve what full RL training accomplishes?
- What makes Effective Rank Acceleration a stable training signal for dual-channel incentives?
- What deployment modes work best for trajectory-aware reward signals?
- When does outcome reward signal become informative during model training?
- Does weight decay directly cause contractive behavior near training examples?
- How should multi-objective post-training balance competing behavioral goals?
- How do you extract reward signals when all rollouts fail?
- Can tree-GRPO work with extremely noisy or sparse outcome reward signals?
- How does active selection of training content differ from random reinforcement sampling?
- What makes content informative and not-yet-mastered for reinforcement during pretraining?
- What distinguishes high-signal prompts from low-signal ones in RL training?
- How does decomposing training telemetry by reward components provide dense feedback?
- Why does binary reward forcing degrade model calibration?
- Why do spurious reward signals improve reasoning for some pretrained models?
- Can log-likelihood loss combined with binary rewards achieve calibration?
- How does negative reinforcement redistribute probability without guiding toward correct answers?
- Can model confidence signals replace explicit external reward functions?
- Can negative feedback through critiques achieve the same steering flexibility as positive preferences?
- How do reward model biases cascade into downstream optimization failures?
- Why do spurious rewards work nearly as well as correct ones?
- What happens when error accumulation and preference signal collapse occur together?
- What makes abstention a learnable behavior instead of a default penalty?
- Why do different models respond differently to spurious rewards?
- Why do spurious rewards work for some models but not others?
- Can log-probability ratios resist reward hacking better than learned PRM signals?
- Can binary judge feedback replace external reward signals for skill learning?
- How does 93% reward reliability compare to other RL noise sources?
- What happens when variance in reward signals comes from a noisy model?
- Are different reward signal sources substitutable in verifier-free RL?
- Can structured rewards still teach models when spurious rewards also work?
- What makes binary rewards more effective than richer reward signals?
- How do internal model mechanisms escape token-level reinforcement signals?
- What makes reward signal sources substitutable across verifier-free RL patterns?
- Can light human signals steer already-learned behavior without preference labels?
- Can checklist-based rewards fix judgment problems in RL training?
- What behavioral changes occur during reward learning training?
- Can meta-reinforcement learning explain why this bias pattern emerges rationally?
- How does reinforcement learning compare to differentiable joint training for RAG?
- Can negative reinforcement alone match full RL performance on domain tasks?
- What makes pretraining composition more important than reward engineering?
- How do reward signals in RLVR interact with pretraining biases?
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- Does reinforcement learning require sufficient pretraining to be effective?
- Can pretraining alone achieve chess performance without reinforcement learning?
- Can importance sampling reduce variance in off-policy reward estimation?
- Do outcome-only reward signals miss step-level errors that compound later?
- What makes process-level supervision better than outcome-only reward signals?
- Is elaborate reward shaping necessary if the pretrained prior already contains good solutions?
- How do process-level rewards compare to environment-extracted next-state signals?
- Why does scalarization of rewards fail for multi-objective GRPO training?
- Can the same variance signal work as both reward and query filter?
- When does a task lack a meaningful multi-dimensional reward structure?
- How does process-based reward differ from outcome-only reward in training?
- Why do outcome-only rewards fail to optimize long-horizon agent behavior?
- What makes current learned reward models fail across different domains?
- How much of LLM behavior traces to post-training reward signals?
- What information do next-state signals contain beyond what scalar rewards capture?
- Should feedback channels be excluded from the reward path in agent evaluations?
- How does forced exploration through diversity rewards differ from suppression-based negative reinforcement?
- How do relational reward signals compare to absolute preference encodings in RL?
- What makes utility-weighted training backfire in machine learning systems?
- What distinguishes inductive inference from negative evidence versus positive patterns?
- Can a rejected-edit buffer work like hard negatives in contrastive learning?
- What makes preventative lessons from failures more valuable than success patterns?
- Why does negative experience transfer better than positive examples alone?
- Can rejected edits serve as negative feedback like hard negatives in contrastive learning?
- Do learning effects explain the drop in red-flagged treatment cases over time?
- Do negative constraints require fundamentally different training signals than positive instructions?
- Why do structure-targeted training negatives fail to fix the underlying problem?
- Can we reverse the instruction-following deficit through targeted training?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does RL improve domain reasoning by adding knowledge or removing it?
When reinforcement learning improves reasoning in specialized domains like medicine, is it teaching models new facts or preventing them from using wrong ones? Understanding this distinction matters for how we design RL training.
directly supports: pruning IS negative reinforcement at the reasoning path level
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
extends: positive reinforcement actively causes entropy collapse; negative reinforcement avoids it
-
Does reinforcement learning update only a small fraction of parameters?
Investigating whether RL algorithms consistently modify only 5–30% of model parameters across different LLMs and RL methods, and what structural properties those sparse updates possess.
complementary: if RL only touches 5-30% of parameters, negative reinforcement may be the primary mechanism for this sparse selection
-
Does the choice of RL algorithm actually matter for reasoning?
Expert Iteration, PPO, and RC-RL show similar performance on reasoning tasks. The question is whether algorithm choice drives results or whether something deeper—like the pretrained model itself—sets the real limits.
supports: if negative reinforcement alone suffices, algorithm choice matters even less
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
- Information-Theoretic Reward Decomposition for Generalizable RLHF
- Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
- A Survey of Reinforcement Learning from Human Feedback
- Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Efficient Reinforcement Learning via Large Language Model-based Search
- Reinforcement Learning with Rubric Anchors
Original note title
negative reinforcement alone matches or exceeds full rl by suppressing incorrect trajectories and redistributing probability mass