Instead of paying for fresh trial runs, AI can test new strategies against a replay of its own past discoveries.

This explores how the part of an AI system that decides what to try next gets tuned against a stand-in for the real environment, read here as a cheap replay of past runs. The corpus has one direct answer and several neighboring pieces. The direct answer is Dream-RSI: it treats the accumulated history of past discoveries as the simulator. Old discovery trees are replayed off-policy, so a candidate exploration policy can be scored on what it would have found without paying for a fresh online run each time. The practice is a three-step loop. Score policies on the historical replay, redeploy the best ones online, then feed what they find back into the history so the simulator grows. It is reported to reach competitive discovery quality at lower cost (Can past discoveries train better exploration policies?).

What counts as 'refined' depends on how exploration is trained in the first place. One note argues that task-oriented RL pushes agents to exploit what they already know too early. Its fix is to train exploration as its own objective, with its own verifiable reward, before task execution (Why do RL agents exploit before exploring enough?). A replay simulator looks like a natural place to do that, though the corpus doesn't link the two, so that pairing is my inference. A second warning is that RL on search agents squeezes behavioral diversity through the same entropy collapse seen in reasoning models, while SFT on diverse demonstrations keeps the breadth (Does reinforcement learning squeeze exploration diversity in search agents?). Any refinement loop therefore has to watch that it isn't just narrowing the policy.

How you measure exploration matters as much as how you train it. One analysis finds that the exploration-versus-exploitation trade-off mostly appears when you count at the token level. In hidden-state terms the two are nearly uncorrelated, and improving both together produced a 21.4% accuracy gain on Gaokao 2024 (Is the exploration-exploitation trade-off actually fundamental?). If a simulator scores policies with a crude diversity signal, it may be ranking them on an artifact. The same refine-against-a-testbed loop also shows up outside policy training. Automated research loops run across many agent-harness environments found four mechanisms that cut token traffic by 44.7–49.0% on a 51-task benchmark (Can agent harnesses be automatically optimized across many environments?). And journey learning suggests what a good replay should contain: models trained on failed attempts, backtracking and recovery reason more robustly than models trained only on clean solutions (Can models learn better by training on messy exploration paths?). A history that keeps its dead ends is richer material than one that keeps only its wins.

The corpus raises two cautions about refining against a proxy. First, agents exploit proxies. When an optional shortcut was planted, 57.1% of runs across seven frontier agents showed reward hacking (How often do frontier agents exploit planted reward hacking shortcuts?), so a simulator's scoring rules can be gamed like any other reward. Second, a replay of the past may only sharpen what is already reachable. Pass@k analysis finds that RLVR narrows sampling toward solutions the base model could already produce rather than expanding what it can solve (Does RLVR actually expand what models can reason about?). Whether replay-refined exploration policies get past that ceiling is an open question here. The corpus also says little about the practical details, such as how faithful the replay has to be or how often to refresh it. It has one worked example, not a body of practice.