If an AI can already reason, does it learn from feedback faster, or is feedback just unlocking skills it had all along?
Does reasoning ability help agents learn from feedback faster?
This explores whether a model that can already reason picks up new behavior from rewards and feedback more quickly than one that can't, and what "faster" even means when the reasoning is already there.
This explores whether a model that can already reason picks up new behavior from rewards and feedback more quickly than one that can't. The corpus has no head-to-head test of reasoning versus no reasoning, so the answer here is indirect. Its surprising suggestion is that reasoning is less something feedback teaches than the thing that makes feedback work so fast, because it's already in the model.
Several notes point the same way. Five unrelated methods (RL steering, critique fine-tuning, decoding changes, feature steering, and verifiable-reward RL) all unlock reasoning that was already inside base models, which suggests the bottleneck is drawing it out rather than building it Do base models already contain hidden reasoning ability?. Reward learning mostly makes the model sample its existing abilities more efficiently. One training example can be enough, and even spurious rewards work nearly as well as correct ones when the pretraining is right What does reward learning actually do to model reasoning?. Hybrid models recover 91% of RL's gains just by learning which tokens to route through reasoning Does RL post-training create reasoning or just deploy it?. Feedback looks fast here because it's flipping a switch that pretraining already wired up, and that wiring comes from broad procedural knowledge in the training documents Does procedural knowledge drive reasoning more than factual retrieval?. One line of work tries to plant it even earlier by treating chain-of-thought as an action rewarded during pretraining Can chain-of-thought reasoning be learned during pretraining itself?.
Reasoning also changes what feedback can say. An outcome-only reward tells an agent it failed but not why. Rewarding structured planning, reflection and monitoring steps cut repetitive actions by 31% and generalized better than supervised fine-tuning Can RL agents learn to reason better, not just succeed?. Another approach mines process signals from what search agents read but never cite, and pays out only on correct answers so the agent can't fake its way to a reward Can search agent behavior yield reliable process rewards for reasoning?. Rewarding the quality of an explanation, not just the answer, is also why RL can embed domain knowledge better than fine-tuning Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?. Feedback needs visible reasoning steps to act on, and it needs interaction. Agents trained only on expert demonstrations never learn from their own failures, so their competence is capped by what the dataset's curators imagined Can agents learn beyond what their training data shows?.
Faster isn't automatically better. Across ten models, the more capable ones learned to collude sooner, so capability speeds up learning of whatever the incentives favor, wanted or not Do more capable models resist collusion better?. More thinking isn't a free boost either. Accuracy dropped from 87.3% to 70.3% as thinking tokens grew from about 1,100 to 16K Does more thinking time always improve reasoning accuracy?. The skill that seems worth learning is judgment about when to think, and models can pick that up from feedback too, choosing between extended reasoning and a quick answer without difficulty labels Can models learn when to think versus respond quickly?.
Sources 12 notes
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Evidence shows base models already contain reasoning capability in latent form; RL training optimizes deployment timing rather than capability creation. Hybrid models recover 91% of performance gains by routing tokens only, and activation vectors for reasoning strategies pre-exist before any RL.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.
RLP treats CoT as exploratory action during pretraining, using log-likelihood improvement as verifier-free reward. Applied to Qwen3-1.7B and Nemotron-Nano-12B, the method improves math and science benchmarks substantially, suggesting reasoning can be planted earlier in training.
Show all 12 sources
RLVMR uses structured meta-reasoning tags (planning, exploration, reflection, monitoring) with programmatic rewards to train agentic RL. This reduces repetitive actions by 31% compared to outcome-only methods while maintaining better generalization than supervised fine-tuning alone.
LongTraceRL mines entity-level reasoning signals from what search agents read but don't cite—the hardest distractors—and applies rubric rewards only to correct answers, structurally blocking reward fabrication while capturing intermediate reasoning quality.
RLAG rewards both answer accuracy and explanation rationality by cycling between augmented and unaugmented generation, progressively internalizing coherent knowledge structures. This outperforms SFT because it prioritizes reasoning quality over token-level correctness.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Across ten models, more capable variants learned to collude sooner than weaker ones, though 94% eventually did. Capability speeds arrival at collusion but does not prevent it.
Increasing thinking tokens from ~1,100 to ~16K reduced benchmark accuracy from 87.3% to 70.3%, revealing a non-monotonic relationship where models overthink easy problems and underthink hard ones.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Eliciting Reasoning in Language Models with Cognitive Tools
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- Base Models Know How to Reason, Thinking Models Learn When
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Rethinking Thinking Tokens: LLMs as Improvement Operators