You can make an AI better by rewriting the code around it instead of retraining it — how do the two differ?
What distinguishes the fast scaffold learning loop from parametric model weight updates?
This explores how improving an AI system by rewriting the code and routing around a frozen model (a scaffold or harness) differs from changing the model's own parameters through training. The corpus has no note that names a 'fast scaffold learning loop', so this answer is assembled from adjacent notes.
This explores how improving an AI system by rewriting the code and routing around a frozen model (a scaffold or harness) differs from changing the model's own parameters through training. No note in the collection names a 'fast scaffold learning loop', so what follows is stitched together from neighbouring material. The corpus doesn't test the two approaches head to head.
The clearest scaffold-side example is a stronger model that built inference-time harnesses for a weaker one. The harnesses nearly doubled the weaker model's Theory-of-Mind scores with no retraining. The gain came mostly from moving unstable reasoning into deterministic code and adding task-specific routing, not from getting the model to think longer (Can a stronger model lift a weaker one at test time without retraining?). Here, what was learned is an artifact: code and routing rules you can read, change or hand to another model. Nothing inside the model moves.
Weight updates behave differently. Reinforcement learning changes only 5-30% of parameters, but the changed set is nearly full-rank and nearly identical across random seeds (Does reinforcement learning update only a small fraction of parameters?). So the change is systematic, but it is spread through the network rather than sitting in one place you can point to. It also can't be cleanly undone. Training on nearly impossible problems teaches degenerate shortcuts that then contaminate capabilities the model already had (Do overly hard RLVR samples actually harm model capabilities?). Low reward variance can push a policy toward generic, input-ignoring templates (Why do language models collapse into generic templates?). Drifting far from the base model can cost the ability to learn later tasks (Does staying close to the base model preserve learning ability?). Weight learning is entangled and sticky: every update touches what came before.
Between these two sits a middle ground that keeps the weights frozen but still adapts the model. Proxy-tuning shifts output distributions at decoding time. It closes 88-91% of the alignment gap and beats direct fine-tuning on knowledge tasks, because direct fine-tuning corrupts knowledge stored in lower layers (Can decoding-time tuning preserve knowledge better than weight fine-tuning?). Representation fine-tuning learns small interventions on hidden states and gets 10-50x better parameter efficiency than LoRA (Can editing hidden representations beat weight updates for finetuning?). Read together, these notes suggest a spectrum: harness code, decoding-time steering, hidden-state edits, then weights. The difference along it is where the learned thing lives and how much existing knowledge it can damage.
Scaffolds have a limit. LLMs can't run iterative numerical methods in latent space. They match a problem to a memorised template and emit plausible but wrong values, and this persists across scale and training approaches (Do large language models actually perform iterative optimization?). That is the kind of weakness a scaffold works around, by putting the loop in deterministic code, as the harness above did. The architecture-side answer is to build the loop into the model. Looped models re-apply layers to gain reasoning through iterated depth instead of size (Can models learn by looping instead of growing larger?). The trade is between a fix you can inspect and swap without touching the model, and a fix that changes what the model can do internally but is harder to isolate and can cost it existing skills.
Sources 9 notes
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
Show all 9 sources
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
ReFT learns task-specific interventions on frozen model representations rather than updating weights, with LoReFT (low-rank linear subspace variant) dramatically outperforming LoRA across reasoning, instruction-following, and NLU benchmarks while using far fewer parameters.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Learning, Fast and Slow: Towards LLMs That Adapt Continually
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- A Mechanistic Analysis of Looped Reasoning Language Models
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models