Why does prompting discover capabilities that need reward-driven refinement?
This explores a puzzle hiding inside two findings the corpus keeps circling: prompting can surface abilities a model already has, yet those same abilities often need reward training to become reliable — so what exactly does each step contribute?
This reads the question as: if prompting and reward training are both just *unlocking* what's already in the base model, why do we need both? The corpus answers with a sharp division of labor — prompting finds the door, reward training learns which door to open every time.
Start with the ceiling. Prompting is powerful but bounded: a single finite transformer is provably Turing-complete given the right prompt Can a single transformer become universally programmable through prompts?, yet that same work notes ordinary training rarely produces models that actually run arbitrary prompted programs. And prompting can only rearrange what's already there — it activates existing knowledge but cannot inject anything absent from training Can prompt optimization teach models knowledge they lack?. So prompting is a search through latent capability, not a way to add capability. Several independent lines confirm the capability is genuinely present to be found: base models already carry latent reasoning that minimal nudging unlocks, via at least five different mechanisms Do base models already contain hidden reasoning ability?.
Here's the catch prompting hits — it's brittle and instance-specific. Zero-shot chain-of-thought only works when the question's meaning flows into the prompt structure before reasoning starts; for simple questions, step-by-step prompting actively hurts Why do some questions perform better without step-by-step reasoning?. Prompting discovers the capability on *some* inputs but can't guarantee the model reaches for it on the right ones. That's precisely the gap reward training closes: RLVR doesn't expand what a model can solve — at high sampling budgets base models actually match or beat it — it sharpens the model toward solutions already in its distribution, raising the odds the latent skill fires on the first try Does RLVR actually expand what models can reason about? What does reward learning actually do to model reasoning?. Prompting reveals the capability exists; reward refinement makes it the default behavior rather than a lucky sample.
The reason refinement needs *reward* specifically — not just more prompting — is that you need a signal for which discovered behavior was good. The corpus shows that signal can come from surprisingly cheap sources: the model's own answer-confidence can rank reasoning traces and improve calibration with no human labels Can model confidence work as a reward signal for reasoning?, and information-theoretic measures can score each step's contribution without annotation Can we reward reasoning steps without human annotation?. Even better signals come from rewards that reason before scoring — generative step-wise judges that meta-reason about reasoning beat classifier-style reward models Can judges that reason about reasoning outperform classifier rewards?, and adding chain-of-thought to the reward model itself raises its ceiling Can reward models benefit from reasoning before scoring?.
The thing you might not have known you wanted to know: this isn't really prompting *versus* reward at all — both are forms of elicitation, just at different time horizons. Prompting is a one-shot search you run at inference; reward training bakes the winning search strategy into the weights so you don't have to find it again every time. The genuinely *new* capability — the kind neither prompting nor RLVR can manufacture — comes only from distillation transferring patterns from a stronger model Does RLVR actually expand what models can reason about?. And the same breadth-vs-depth logic recurs even within a single inference pass: allocating compute to diverse abstractions explores better than hammering one reasoning chain deeper Can abstractions guide exploration better than depth alone?. Discovery and refinement are the same act at different scales.
Sources 11 notes
Research proves a single finite-size transformer exists that can compute any computable function given the right prompt, achieving complexity bounds nearly matching unbounded models. However, standard training rarely produces models that learn to implement arbitrary programs this way.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Saliency analysis reveals that CoT prompting fails when question information doesn't aggregate into the prompt structure before reasoning begins. For simple questions, direct question-to-answer flow outperforms step-by-step reasoning, showing the optimal prompt depends on question type, not just task category.
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
L2T uses PAC-Bayes bounds and Fisher information to compute per-episode rewards measuring each step's contribution to correctness. This annotation-free approach matches dense feedback quality while eliminating the cost of outcome-only methods that produce 2x excess tokens.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
RLAD jointly trains abstraction and solution generators, showing that allocating test-time compute to diverse abstractions outperforms parallel solution sampling at large budgets. Abstractions create structured breadth-first exploration that prevents the underthinking failure mode of depth-only reasoning chains.