What makes session-aware multi-turn tracking necessary for asynchronous training?
This explores why asynchronous RL — where text generation runs continuously while the model trains on a lagging version of itself — forces you to track the full state of a multi-turn session rather than scoring isolated steps.
This explores why asynchronous RL — where generation and training are decoupled so workers keep producing rollouts while the model updates on a slightly older version — makes per-session, multi-turn tracking unavoidable. The short answer the corpus points to: once you stop waiting for each rollout to finish before learning, the unit of learning stops being a single response and becomes a whole trajectory, and trajectories only make sense if you keep their session state intact.
Start with what asynchrony actually changes. In a fully asynchronous setup, generation never pauses; training consumes samples that were produced by mixed, sometimes stale model versions Can RL training run while generation continues without waiting?. That staleness is tolerable for a one-shot answer, but in a multi-turn task a later turn depends on earlier ones — so you can't reconstruct what a reward means unless you carry the session's history with the sample. The same work notes this is precisely what makes *multi-turn* RL practical, which is a tell: the engineering problem async creates and the problem session-tracking solves are the same problem.
The long-horizon papers show why the state you track has to be the session, not just the step. When an agent does iterative search or software work, reasoning spent early in a turn eats the context budget needed to absorb evidence from later turns, degrading the whole episode unless you budget per turn across the session Does limiting reasoning per turn improve multi-turn search quality?. And RL only scales to these stateful, multi-step environments — doubling SWE-bench performance — because it treats the delayed reward as attaching to a sequence of dependent actions, not a single move Can reinforcement learning scale beyond single-turn language tasks?. A reward that only arrives at the end is uninterpretable if you've thrown away the turns that earned it.
Here's the part you might not expect: keeping the trajectory around isn't just bookkeeping, it's where the training signal comes from. You can derive dense, per-step rewards directly from the *structure* of a trajectory — tree topology, tool-call positions, expert-aligned actions — instead of training a separate reward model Can trajectory structure replace hand-annotated process rewards?. And how you store each session matters: treating successful episodes as concrete demonstrations and failures as abstracted lessons gives better learning while spending far less context than dumping every turn uniformly Should successful and failed episodes be processed differently?. Even in-context learning shows the same dependency — models generalize across sequential decision tasks only when given full or partial trajectories from the same session, not isolated examples Why do trajectories matter more than individual examples for in-context learning?.
So the necessity is really a chain: asynchrony breaks the one-rollout-at-a-time assumption, which makes the trajectory the natural learning unit, which means the reward signal, the context budget, and the memory format all become session-scoped quantities. Drop the session tracking and a stale async sample is just a pile of orphaned turns with a number attached to it.
Sources 6 notes
AReaL enables continuous generation across workers while training runs on mixed model versions using modified PPO. The system achieves high GPU utilization and handles stale samples effectively, making multi-turn RL practical.
Unrestricted reasoning within single search turns consumes context needed for subsequent retrieval rounds, degrading the agent's ability to incorporate new evidence. Setting per-turn reasoning budgets, not just overall time limits, prevents this context erosion and maintains search quality across iterations.
Modified DAPO training doubled SWE-bench Verified performance from 20% to 39% on Qwen2.5-72B, matching larger models. This demonstrates RL works in stateful multi-step environments with delayed rewards and complex feedback, beyond theoretical single-turn MDPs.
Tree-GRPO, Supervised RL, and ToolPO each convert sparse outcome rewards into dense step signals by exploiting different structural features—tree topology, expert-aligned actions, and tool-call positions—eliminating the need for annotated process reward models.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
In-context learning for sequential decision-making requires full or partial trajectories from the same environment level, not just isolated examples. This structural property—trajectory burstiness—allows models to generalize across vastly different tasks without weight updates.