SYNTHESIS NOTE
Topics›Reinforcement Learning›this note

Can natural language feedback overcome numerical reward plateaus?

Exploring whether chain-of-thought critiques can push past performance ceilings that scaling data alone cannot break in reinforcement learning for reasoning tasks.

Synthesis note · 2026-02-22 · sourced from Reinforcement Learning
How should we spend compute at inference time?

Three failure modes of purely numerical RL for reasoning: (1) performance plateaus despite 8x scaling of training examples (from 4k to 32k); (2) self-reflection behaviors during RL, often celebrated as "aha moments," contribute minimally to successful problem-solving; (3) persistent failures on certain problems despite extensive trial-and-error training. The common cause: numerical feedback contains limited information about WHY a response is correct or incorrect and HOW to improve.

Critique-GRPO demonstrates that RL-finetuned models, even after exhibiting performance plateaus, can generate correct refinements on persistently failed problems when provided with chain-of-thought critiques. The key is integrating both natural language feedback (NLF) and numerical feedback within online RL. The model learns from initial responses and critique-guided refinements simultaneously while maintaining exploration.

This is significant because it challenges the implicit assumption that RL's learning signal is sufficient for arbitrarily complex reasoning. Since Does reflection in reasoning models actually correct errors?, the ineffectiveness of self-reflection during RL training is predictable — the model cannot generate useful critiques of its own failures. External critiques break the ceiling because they provide the information that numerical rewards lack: specific identification of where reasoning went wrong.

The practical architecture has three components: (1) the model generates initial responses; (2) a reasoning-based reward model generates CoT critiques identifying flaws; (3) a shaping function enhances learning from valid refinements and heavily penalizes failed refinements. This approach encourages the model to integrate targeted refinements while preserving exploration.

Since Do critique models improve diversity during training itself?, the NLF mechanism works by expanding the effective exploration space — critiques point toward regions of solution space that numerical rewards cannot identify.

Semantic reward shaping as lightweight NLF: The Semantic Reward Shaping paper proposes a complementary mechanism: using a small encoder-only transformer to compute cosine similarity between generated explanations and ground-truth references. This provides a dense, semantically rich reward signal within GRPO — not as information-rich as full CoT critiques, but vastly cheaper and faster than LLM-as-judge evaluation. The approach combines semantic similarity reward with auxiliary correctness and formatting rewards, significantly improving explanation faithfulness over SFT baselines. This occupies a middle ground between brittle keyword metrics (ROUGE) and expensive LLM-based critiques — suggesting the NLF principle scales down to lightweight implementations when full CoT critique is impractical.

Textual gradients as generalized NLF: TextGrad (2406.07496) formalizes the broader principle: natural language criticism can serve as "textual gradients" propagated through arbitrary computation graphs including LLM API calls, simulators, and external solvers. Each AI system component is a node in a computation graph; textual feedback describes how variables should change to improve the system. This extends NLF from RL plateau-breaking to general AI system optimization — the same principle (informative language feedback > scalar signal) applies at the system level, not just the training level.

Inquiring lines that read this note 171

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Why do reward structures fail to shape long-term agent learning? How do self-generated feedback mechanisms enable effective model learning? Can alternative training methods improve on supervised fine-tuning for language models? How do we evaluate AI systems when user perception misleads actual performance? What properties determine whether reward signals teach genuine reasoning? When do additional thinking tokens stop improving reasoning performance? What constrains reinforcement learning's ability to expand model reasoning? How can real-time alliance measurement improve therapy outcomes? Why do LLM chatbots fail as independent therapeutic agents? Can ensemble evaluation methods reduce bias more than single judges? What pretraining choices and baseline capability constrain reinforcement learning gains? How should iterative research systems allocate reasoning per search step? How does policy entropy collapse constrain reasoning-focused reinforcement learning? Can language model RL training avoid reward hacking and misalignment? Does decoupling planning from execution improve multi-step reasoning accuracy? How does AI assistance affect human cognitive development and reasoning autonomy? Can prompting strategies overcome LLM biases without model fine-tuning? How can process reward models supervise complex reasoning traces? Do autonomous architecture discoveries follow predictable scaling laws? How does example difficulty affect learning efficiency in language models? Does reinforcement learning teach reasoning or just when to reason? How does objective evolution guide discovery better than fixed planning? How do LLMs distinguish causal reasoning from temporal and semantic associations? How can we distinguish genuine user preferences from measurement artifacts? Which computational strategies best support reasoning in language models? How does latent reasoning compare to verbalized chain-of-thought? Can self-supervised signals enable process supervision without human annotation? Does RLHF training sacrifice accuracy and grounding for user agreement? Why do benchmark improvements fail to reflect actual reasoning quality? What structural biases does transformer attention create in language model outputs? Can prompting inject entirely new knowledge into language models? Can AI systems balance emotional competence with factual reliability? Why does supervised fine-tuning improve accuracy while degrading reasoning quality? How can AI agents autonomously learn and transfer skills across tasks? How should agents balance memory condensation to optimize context efficiency? How do policy learning algorithm choices affect multi-objective optimization stability? What determines success in training models on multiple tasks? How do aggregate reward models systematically exclude minority user preferences? What makes weaker teacher models effective for stronger student training? How can AI systems learn from failures without cascading errors? How should inference compute be adaptively allocated based on prompt difficulty? Why does reinforcement learning suppress output diversity compared to supervised fine-tuning? Does externalizing cognitive work and state improve agent reliability? How should human oversight be integrated with autonomous AI systems? Can single-axis benchmarks accurately predict agent deployment success? How do training data properties shape reasoning capability development?

Related concepts in this collection 4

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
17 direct connections · 130 in 2-hop network ·medium cluster Open in graph ↗
Can natural language feedback overcome numerical… Do critique models improve diversity during traini… Does reflection in reasoning models actually corre… Does revising your own reasoning actually help or … Does policy entropy collapse limit reasoning perfo…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Why does self-reflection during training fail to improve model self-correction?How do semantic reward shaping approaches compare to full critique models?Can textual gradients generalize natural language feedback across computation graphs?What information do numerical rewards fail to provide for reasoning tasks?
Original note title

natural language feedback breaks rl performance plateaus that scaling numerical rewards alone cannot resolve