If an AI is tested against a replay of past attempts, can that replay judge paths nobody ever tried?

This asks whether a replay simulator built from past exploration can still fairly score a new policy once that policy wanders into branches nobody has tried. The corpus has no direct measurement of how replay fidelity decays off the recorded path. But several notes point the same way: replay is trustworthy only where history actually went. Dream-RSI scores exploration policies by replaying accumulated discovery trees off-policy instead of running every candidate online, reportedly with competitive discovery quality at lower cost Can past discoveries train better exploration policies?. Its loop doesn't stop at replay, though. It evaluates on history, redeploys online, then expands the simulator. That last step looks like an acknowledgment that new branches get a score only after someone has walked them. That reading is my inference, not something the note states.

The reason is simple. On a branch that isn't in the record, a replay has nothing to replay, so it has to guess or stay silent. The closest parallel in the collection is about model updates. An update's effect depends on the context it came from: the parent model, the data, the training stage, the evaluation criteria. So past success shouldn't be treated as permission to reuse it. The note recommends applicability checks and bounded trials instead, and warns that promoting a child rewrites the parent that future evidence is measured against Should past update success guide future model changes?. Replay has the same problem. History is evidence about the situation that produced it, and a policy that changes where you go also changes which situation you're in.

There's also a shape-match worth holding next to this. RLVR analyses find that base models beat RLVR-trained ones at high sample counts, which suggests training sharpens sampling toward answers already in the base distribution rather than reaching new ones Does RLVR actually expand what models can reason about?. It isn't a replay result. But a policy tuned against a record of past exploration faces a similar pull: staying on well-trodden ground is what can be measured, and stepping off it can't be scored yet.

A third issue is that a new branch produces a new sequence, not just new steps. Sequences of individually permissible actions can collectively break constraints that no single step breaks agent-security-is-a-property-of-behavior-over-time-rather-of-individual-act, and per-action checks structurally can't state constraints that depend on prior history Can stateless checks ever catch sequence-level constraint violations?. A replay that matches a novel path against recorded steps one at a time could pass steps that are fine locally and miss what the whole trajectory does.

So the corpus's implied answer is that fidelity doesn't carry over automatically. It holds where history has coverage and has to be re-earned elsewhere. The practical moves it suggests are to keep what was replayed separate from what was extrapolated, and to run bounded live trials on the new branches. Claims about a run should also rest on recorded evidence of how it went, not just a final number. BenchShield's approach is to certify whether an agent followed the intended path using infrastructure evidence rather than terminal scores alone Can infrastructure evidence replace terminal scores in benchmark validation?. Trace-grounded reports are good at flagging unsupported claims and evidence gaps Can execution traces ground honest explanations of agent behavior?. What the collection doesn't yet say is how far off-history a policy can go before the replay score stops meaning anything.