Can in-context learning replicate the timing effects that RL teaches models?
This explores whether in-context learning — steering a model at inference time with examples and no weight updates — can reproduce the sequence- and phase-dependent effects that reinforcement learning bakes in during training.
This explores whether in-context learning can stand in for RL's timing effects, and the corpus splits the question into two halves that point in opposite directions. The first half is encouraging. A recurring finding is that RL doesn't teach models genuinely new reasoning — it surfaces strategies already latent in pretraining. Verifiable-reward RL acts as a catalyst that improves sampling efficiency within existing capability boundaries rather than expanding them, to the point that a single example or even spurious rewards can trigger the gains What does reward learning actually do to model reasoning? How does RL training reshape reasoning and what gets lost?. If RL is mostly activation rather than construction, then a lot of what it 'teaches' is in principle reachable without weight updates at all — you just need the right context to call the same priors forward.
And in-context learning can in fact reach temporal, sequential behavior. The trajectory-burstiness work shows models learning sequential decision-making purely in context — but only when the context contains full or partial trajectories from the same environment, not isolated examples Why do trajectories matter more than individual examples for in-context learning?. That's a striking parallel: timing structure can be conveyed through the *shape* of the prompt, not just through gradient steps. The same flavor shows up in how instruction tuning works — what transfers is knowledge of the output space and format, not semantic task understanding, since models trained on deliberately wrong instructions perform about as well Does instruction tuning teach task understanding or output format?. Much of what looks like 'learning' is really format and distribution activation, exactly the kind of thing a well-built context can supply.
The second half is where replication breaks down. RL's most interesting timing effects are properties of the *training trajectory itself*, and those have no in-context analog. RL training moves through a two-phase dynamic — first execution correctness drives gains, then strategic planning becomes the bottleneck, with planning-token entropy rising while execution entropy stabilizes Does RL training follow a predictable two-phase learning sequence?. The *order* in which you train domains mechanically reshapes entropy: training structured tasks before creative ones prevents entropy collapse from wrecking open-ended ability, worth a measurable margin over joint training Does training order reshape how models handle different task types?. These are path-dependent consolidation effects — what got consolidated first constrains what can be learned later. A prompt has no first-and-then; it presents everything at once.
There's also a structural reason in-context steering can't fully impersonate RL: RL leaves a physical fingerprint. It updates only 5–30% of parameters, in sparse but near-full-rank subnetworks that are nearly identical across random seeds — structural, repeatable surgery on the weights Does reinforcement learning update only a small fraction of parameters?. It also converges the model onto a single dominant pretraining format while suppressing alternatives within the first epoch Does RL training collapse format diversity in pretrained models?. In-context learning leaves the weights untouched, so it can't make a behavior *default* the way RL does — it can only invoke it for the length of the context. Effects like calibration are a clean example: binary-reward RL provably degrades calibration and the fix is a second reward term, a training-time intervention with no obvious prompt equivalent Does binary reward training hurt model calibration?.
So the honest answer the corpus suggests: in-context learning can replicate the *capability* timing effects — surfacing latent sequential and strategic behavior, since RL was largely activating those anyway — but not the *process* timing effects. The two-phase consolidation, the order-dependent entropy dynamics, and the sparse durable rewiring are artifacts of the learning trajectory, and a context window has no trajectory to speak of. What you didn't know you wanted to know: the question quietly conflates two different things RL does, and the corpus pulls them cleanly apart — one of them ICL can fake, the other it structurally cannot.
Sources 9 notes
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Research shows that verifiable rewards act as catalysts that surface existing capabilities from pretraining, not teachers that build new reasoning. RL updates are structurally sparse and bounded by the pretrained prior, not algorithmic sophistication.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.