When an AI improves by tweaking its surrounding code instead of its brain, how can it still overfit?
Why is the fast non-parametric loop vulnerable to overfitting differently than model weights?
This explores how a system that adapts without touching its weights (the fast loop: harness code, routing, iterated computation at inference time) can overfit in a different way from a system whose weights are updated by gradient training.
This explores how adapting a system outside its weights, through harness code, routing rules, or extra passes at inference time, can overfit in a different way from changing the weights themselves. The corpus has no note that compares the two head-to-head, and none that tests overfitting in memory or retrieval loops directly. What it does have is a detailed picture of how weight-side overfitting works, plus a few examples of the fast side. The contrast between them is informative.
When weights overfit, the problem gets baked in and spreads. Training on nearly impossible RLVR problems teaches models degenerate shortcuts, and those shortcuts then contaminate capabilities the model already had (Do overly hard RLVR samples actually harm model capabilities?). When reward variance is low, policies collapse into generic, input-ignoring templates (Why do language models collapse into generic templates?). RL also tends to lock onto a single format from pretraining and suppress the others (Does RL training collapse format diversity in pretrained models?). Part of why this sticks is that RL's changes are structured: it touches only 5–30% of parameters, but the updates are nearly full-rank and nearly identical across random seeds (Does reinforcement learning update only a small fraction of parameters?). A weight-side failure is diffuse, hard to localize, and hard to undo.
The fast loop overfits in the opposite way: it fits the shape of the task rather than warping the model. The clearest example is a stronger model building an inference harness that nearly doubled a weaker model's Theory-of-Mind scores. It did so mainly by moving unstable reasoning into deterministic code and adding task-specific routing (Can a stronger model lift a weaker one at test time without retraining?). Nothing in the weaker model changed. But rules and routing written against one benchmark encode that benchmark's quirks, so the overfit lives in the scaffolding. That is my inference rather than something the note tests. If it holds, the damage is local and inspectable: you can read the routing rule, see what it assumes, and delete it. A fast loop also gets many more rounds of adjustment against the same feedback, so the risk is fitting the evaluation itself, not distorting the model.
Some notes describe the middle ground. Utility-weighted training harms representation learning because the training signal is bent toward decisions. Training with a symmetric loss and adjusting predictions afterward beats optimizing the utility directly (Can utility-weighted training loss actually harm model performance?). That is a case for keeping the slow-learned part clean and putting task-specific adaptation in a separate layer on top. Representation finetuning does something similar by intervening on frozen hidden states, reportedly with 10–50x better parameter efficiency than LoRA (Can editing hidden representations beat weight updates for finetuning?). Looped models put the adaptation into extra passes through shared layers, with weights fixed. Their convergence signals give a natural point to stop (Can models learn by looping instead of growing larger?), and spectral-norm constraints give formal stability guarantees (Can looped computation replace parameter count in world models?). Weight training has no equivalent built-in stopping signal for overfitting.
So the corpus points to two different failure shapes. Slow weight updates overfit by absorbing shortcuts and collapsing diversity, and the result is durable and entangled with everything else the model knows. Fast non-parametric loops overfit by hard-coding assumptions about the task, and the result is shallow and removable, but easy to trust because it scores well on the benchmark it was tuned on. Whether memory and retrieval loops behave like the harness case is an open gap in this collection.
Sources 9 notes
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Show all 9 sources
Asymmetric loss functions correctly incentivize choosing but degrade representation learning by reducing gradient signals for substantive feature acquisition. Training with symmetric loss then adjusting predictions post-hoc outperforms direct utility-weighted training on the same utility objective.
ReFT learns task-specific interventions on frozen model representations rather than updating weights, with LoReFT (low-rank linear subspace variant) dramatically outperforming LoRA across reasoning, instruction-following, and NLU benchmarks while using far fewer parameters.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
LoopWM achieves up to 100x parameter efficiency by refining latent environment states through iterative computation in a shared block, with spectral-norm constraints providing formal stability guarantees. The approach mirrors physical system recurrence, spending more depth on harder prediction steps.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- Loop the Loopies!
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- A Mechanistic Analysis of Looped Reasoning Language Models
- Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?