If you train an AI against test cases you wrote yourself, does it learn the real task or just how to pass?
How do planted cases perform inside an optimizer loop as training signals?
This explores what happens when deliberately inserted test cases with known answers ("planted" cases) are used as the scoring signal inside a training or optimization loop. The corpus has no note on planted cases directly, so I'm reading the question through its notes on training signals inside optimizer loops.
This reads the question as: if you seed a training loop with cases you built and know the answers to, do they teach the model what you want? The corpus has nothing that tests this head-on, so what follows is inference from neighboring notes, not a documented result.
The closest match is a note on reward hacking. It finds one shared mechanism across weight training, output selection, and prompt revision: the optimizer is pushed against a signal that only partly represents the real task (Does reward hacking always stem from the same failure?). A planted case is exactly that kind of signal, a small stand-in for the task. So how well it works depends less on the case itself than on the gap between passing it and doing the real job. A loop that gets rewarded for passing will find whatever passes.
Other notes show where training signals usually come from, and planted cases are a third source. Deployment can harvest them: every user reply, tool output, or error is a next-state signal that can train the policy (Can agent deployment itself generate training signals automatically?). Search can derive them: tree-search outcomes rank solution paths and stand in for human labels (Can tree search replace human feedback in LLM training?). Planted cases are designed rather than found. That gives you a known answer and control over coverage, but they only cover the situations you thought to plant.
What the optimizer learns from a fixed set of cases is a real risk. Supervised fine-tuning on optimization problems improved how outputs looked (valid structure, expected sections) without making them feasible (Does supervised fine-tuning actually improve reasoning on optimization problems?). RL-tuned models dropped sharply on slightly changed variants of problems they had trained on, which points to template-matching rather than a learned procedure (Do fine-tuned language models actually learn optimization procedures?). A planted set with one right answer per case invites the same shortcut. The practical check is to hold back varied versions of each case and score those.
There is also a side effect on diversity. Rewarding only final-answer correctness narrows the policy, and the narrowing spreads to problems the model hasn't solved yet (Does outcome-based RL diversity loss spread across unsolved problems?). If the loop feeds into search later, training that keeps varied competent answers can beat training on one scalar score (Should training maximize diversity when models feed into search?). Planted cases scored pass/fail could push toward the collapsed policy that note warns about.
Sources 7 notes
Reward hacking arises during weight training, output selection, and prompt revision through a shared failure: optimization against signals that incompletely represent the actual task. The substrate matters less than the misalignment between the scoring function and ground truth.
Every agent action produces a next-state signal (user reply, tool output, error, GUI change) that can train the policy directly. This universal signal source eliminates the need for separate training datasets across conversations, terminal tasks, SWE, and tool use.
AlphaLLM uses tree search outcomes and three critic models to derive dense reward signals equivalent to human-labeled feedback. Tree structure naturally ranks solution paths by success, replacing the annotation oracle that standard RLHF requires.
Supervised fine-tuning makes model outputs look correct—proper JSON structure, valid identifiers, expected sections—without making them physically feasible. The model learns surface features of solutions, not the reasoning to construct valid ones.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Show all 7 sources
RL that rewards only final answer correctness sharpens the policy globally, concentrating probability mass on correct trajectories for solved problems while simultaneously reducing diversity on unsolved ones. Historical exploration (training diversity via UCB-style bonuses) and batch exploration (test-time diversity via repetition penalties) require structurally different mechanisms.
Vector Policy Optimization trains models to emit varied competent solutions rather than converging to one answer. This unlocks search procedures like evolutionary algorithms to explore and combine modes, solving problems that entropy-collapsed policies cannot reach at all.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Vector Policy Optimization: Training for Diversity Improves Test-Time Search
- Outcome-based Exploration for LLM Reasoning
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- Jointly Reinforcing Diversity and Quality in Language Model Generations
- Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing