If an AI is rewarded on the same problems it's training on, can that reward tell real understanding from memorized answers?
How much does same-batch reward signal confuse memorization with generalization?
This explores whether a reward computed on the same batch of examples the model is training on can tell a model that memorized answers from one that learned to reason. The corpus has no note that measures this directly, so what follows is triangulated from related work.
This explores whether a reward computed on the same batch of examples the model is training on can tell a model that memorized answers from one that learned to reason. Nothing in the collection puts a number on that, so there's no honest 'how much.' But several notes show that a reward is a blunt instrument for this distinction. The clearest case is What does reward learning actually do to model reasoning?. RLVR seems to activate strategies the model already had from pretraining rather than teach new ones. A single training example is enough to trigger the gain, and spurious rewards work nearly as well as correct ones for models with the right pretraining. If a reward that is partly random still produces the improvement, the reward isn't measuring whether anything new was learned. It says an output scored, not why it was right.
A second note shows how easily memorization passes for generalization. In Does multimodal zero-shot performance actually generalize or interpolate?, zero-shot performance tracks how often a concept appeared in pretraining, and gains need exponentially more data. Apparent generalization was largely frequency. A batch-level reward has the same blind spot. A correct answer earns credit whether it came from recall or reasoning. [[reward-hacking-can-arise-when-weights-are-updated-when-outputs-are-selected-and]} explains why this keeps recurring. Reward hacking, across weight training, output selection and prompt revision, comes from optimizing against a signal that only partly represents the task. Memorization is a shortcut that satisfies such a signal without doing the task.
The corpus's fixes all add information to the reward rather than trusting one score. Can breaking down instructions into checklists improve AI reward signals? splits quality into verifiable sub-criteria, which cuts overfitting to superficial artifacts. Can judges that reason about reasoning outperform classifier rewards? judges the reasoning steps instead of just the outcome. Can natural language feedback overcome numerical reward plateaus? argues that numerical rewards leave out why a failure happened, and a written critique unlocks solutions the model was stuck on. Can peer models replace external judges for reward signals? attacks the other half. When a model grades itself, its bias and collapse feed back into training, and a diverse cohort of peer models breaks that loop.
Two lateral framings suggest that mixing the two isn't the real problem. It's forcing both through one channel. Can one model memorize and generalize better than two? gives memorization its own component: cross-product features memorize rare cases while embeddings generalize, and each stays small because the other covers its weak spot. Should successful and failed episodes be processed differently? does something similar for RL by keeping successes as concrete demonstrations and failures as abstracted lessons. Do RL agents accidentally use environments as memory? adds a caution. Agents trained with a plain reward develop memory-like behavior nobody asked for, so a scalar reward can't tell you which mechanism is producing the score.
The takeaway is that the corpus supports 'quite a lot, and mostly invisibly.' It doesn't isolate same-batch effects, and no note tests memorization against generalization inside a single RL batch. That would be a real gap to explore.
Sources 10 notes
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Across 34 models and 5 datasets, multimodal models require exponentially more pretraining data for linear performance gains on downstream tasks. Performance correlates with how often test concepts appeared during pretraining, not genuine generalization ability.
Reward hacking arises during weight training, output selection, and prompt revision through a shared failure: optimization against signals that incompletely represent the actual task. The substrate matters less than the misalignment between the scoring function and ground truth.
RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
Show all 10 sources
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Co-RL trains decoupled models using peer predictions as rewards, avoiding the bias and collapse of self-generated feedback. Heterogeneous cohorts consistently improve reasoning across benchmarks and often match ground-truth supervised training.
Wide & Deep models train memorization (cross-product features) and generalization (embeddings) together, allowing each component to specialize: the wide part becomes small because deep handles common cases, and deep doesn't overfit rare items because wide captures them. Ensembling requires both halves full-size.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Mathematical proof shows that environmental artifacts reduce information needed to represent history in RL agents. Path-following agents naturally develop memory-like behavior through standard reward optimization, satisfying situated cognition criteria without explicit memory objectives.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Reward Reasoning Model
- Artifacts as Memory Beyond the Agent Boundary
- RM-R1: Reward Modeling as Reasoning
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Useful Memories Become Faulty When Continuously Updated by LLMs
- AgentFly: Fine-tuning LLM Agents without Fine-tuning LLMs
- SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains