INQUIRING LINE
Inquiring lines›How should we train models for cap…›How do attention and architecture…›What constrains reinforcement lear…›this inquiring line

In AI training, the most useful questions aren't the hardest — they're the ones the model answers correctly only sometimes.

What distinguishes high-signal prompts from low-signal ones in RL training?

This explores what actually makes a training prompt useful in reinforcement learning — and the corpus's answer is surprisingly consistent: it's not difficulty or correctness, it's the *variance* of outcomes a prompt produces across repeated attempts.

This explores what makes a prompt worth training on in RL, and the corpus converges on a single, non-obvious answer: signal lives in variance, not correctness. A high-signal prompt is one where the model, sampled several times, produces a *spread* of outcomes — some good, some bad. That spread is what lets the learning algorithm tell better trajectories from worse ones. When every rollout for a prompt looks the same — all correct or all wrong — there's nothing to compare, the task gradient collapses, and regularization takes over and drags the policy toward generic, input-agnostic templates Why do language models collapse into generic templates?. The practical upshot is a filter: select high-variance prompts *before* you update, and performance recovers across tasks and scales.

Strikingly, the same statistic does double duty. Cross-rollout variance can simultaneously weight which tokens matter within a prompt and decide which prompts to keep at all — discarding the degenerate, no-information queries where every rollout agrees — and reusing one self-supervised measure at both levels buys faster, more stable training, especially on tasks with no clean verifier Can one statistical measure serve dual purposes in RL training?. So 'high-signal' isn't a property you attach by hand; it's something the rollouts reveal.

But variance alone isn't the whole story — there's a failure mode at the hard end that looks like signal and isn't. Prompts that are nearly impossible do produce rare, scattered successes, and group-relative normalization treats those lucky hits as huge-advantage trajectories. The model then learns the *shortcut* that happened to work — answer repetition, skipping computation — rather than real reasoning, and that damage bleeds into capabilities it already had Do overly hard RLVR samples actually harm model capabilities?. So the sweet spot is prompts with genuine outcome spread the model can actually resolve, not prompts so hard that the only 'signal' is noise being amplified.

There's also a ceiling worth naming: no prompt, however well-chosen, injects knowledge the model doesn't have. Optimization pressure only surfaces what pretraining already laid down — which is why spurious or even random rewards boost some models dramatically and do nothing for others; the reward isn't teaching, it's activating latent behavior that a particular pretraining recipe made available Why do random rewards improve reasoning for some models but not others?, much as prompt optimization can reorganize existing knowledge but never add new Can prompt optimization teach models knowledge they lack?. High-signal prompts are levers on capability that's already present.

The deeper lesson is that 'signal' is relational, not absolute — it's a property of the prompt-and-model pair at a moment in training. What carries signal shifts as the model learns: early RL is driven by execution correctness, then the bottleneck moves to strategic planning, so the prompts that teach the most change phase to phase Does RL training follow a predictable two-phase learning sequence?. And even the shape of the signal you supply matters — training on negative examples alone can match full RL while preserving diversity, whereas piling on positive reinforcement concentrates probability mass and quietly narrows the model Does negative reinforcement alone outperform full reinforcement learning?. High-signal prompting, in the end, is less about picking hard questions and more about feeding the model comparisons it can currently learn from.

Related lines of inquiry

Explore this area

Sources 7 notes

Why do language models collapse into generic templates?

When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.

Can one statistical measure serve dual purposes in RL training?

DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.

Do overly hard RLVR samples actually harm model capabilities?

Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.

Why do random rewards improve reasoning for some models but not others?

Qwen2.5-Math gains 16-25% MATH-500 improvement from random or incorrect rewards by activating latent code-reasoning behavior from pretraining, while Llama and OLMo show no gains. Pretraining format determines what optimization pressure can surface.

Can prompt optimization teach models knowledge they lack?

Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.

Show all 7 sources
Does RL training follow a predictable two-phase learning sequence?

Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.

Does negative reinforcement alone outperform full reinforcement learning?

Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.

Research prompt for your LLMexpand ↓

Copy into ChatGPT or Claude to take this line of inquiry further — it asks the model to find newer work and re-test which earlier constraints still hold.

You are an RL-training analyst. Still-open question: what distinguishes high-signal prompts from low-signal ones in RL post-training of LLMs?

What a curated library found — and when (dated claims, not current truth; findings span ~2023–2026):
- Signal lives in variance, not correctness: a high-signal prompt yields a *spread* of outcomes; when all rollouts agree (all right or all wrong) the gradient collapses and regularization drags the policy toward input-agnostic templates (~2025).
- Cross-rollout variance does double duty — weighting which tokens matter AND filtering which prompts to keep — one self-supervised measure buying faster, stabler training where no clean verifier exists (~2025).
- At the hard end, near-impossible prompts give rare lucky hits that group-relative normalization treats as huge-advantage, so the model learns shortcuts that damage existing capability (~2026).
- Ceiling: no prompt injects new knowledge — spurious/random rewards boost some models sharply and do nothing for others; reward activates latent pretraining behavior (~2025).
- Signal is relational, shifting phase-to-phase (execution correctness early, planning later); negative-only reinforcement can match full RL while preserving diversity, whereas positive reward narrows the model (~2025).

Anchor papers (verify; mind their dates): Echo Chamber: RL Post-training Amplifies Pretraining (2504.07912, 2025); The Surprising Effectiveness of Negative Reinforcement (2506.01347, 2025); The Art of Scaling RL Compute (2510.13786, 2025); Mechanistically Interpreting Sample Difficulty in RLVR (2605.28388, 2026).

Your task: (1) RE-TEST EACH CONSTRAINT — judge whether newer models, training, tooling, orchestration, or evaluation have RELAXED or OVERTURNED each finding; separate the durable question from the perishable limitation, cite what resolved it, and note where a constraint still holds. (2) Tension-surfacing: surface the strongest CONTRADICTING work from the last ~6 months, foregrounding where sources DISAGREE (variance-filtering vs. hard-prompt curricula; positive vs. negative reinforcement). (3) Propose 2 questions that ASSUME the regime moved. Cite arXiv IDs; flag anything you cannot ground in a real paper.