Does fixed dialogue history bias role-play agent evaluation?

The paper argues that the standard role-playing agent benchmark, which has an RPA "continue a fixed dialogue history" and scores the continuation "using a fixed rubric detached from the user," has two limits. The abstract says the authors "identify and empirically demonstrate" both, though the excerpt does not show that evidence. The first limit is that an RPA's output "is shaped by the preceding dialogue history," so the score is not a clean reading of role-playing ability in a real multi-turn setting. The second is that "user experience varies substantially across individuals," so a fixed rubric "need not align with user satisfaction." The discussion puts the first as placing an RPA "in a trajectory that it did not help construct," and the second as compressing "genuine individual differences in satisfaction into a single standard."

The proposed fix is PALATE, a benchmark built on user simulators. Each simulator is trained from a real user's history and generates free multi-turn dialogue with a candidate RPA. Evaluation runs on three tracks: personalized, generic turn-level, and whole-session. The paper splits one person into "a behavior policy learned from real dialogue and an individual utility supervised by experience annotations," which "changes the basic unit of role-play evaluation from an isolated RPA to a user-RPA pair." The main evaluation uses five per-user simulators over a pre-frozen panel of character profiles, drawn from a pool of 300. Its headline observation is that across 16 candidates "advantages on the three tracks do not coincide, and the five users do not share a single best candidate." The authors conclude that the central output is "an interactive evaluation profile," not "another scalar leaderboard."

This sits close to Can one persona population evaluate different application types?, which also puts simulated users in front of the system under test. That note's simulated users come from existing persona datasets, and it defers fidelity to real users as future calibration. PALATE instead ties each simulator to one real user's history and a separate satisfaction utility, which is a different answer to the same gap, although the excerpt reports no check of simulator fidelity. Conditioning on real dialogue rather than a written description is also the direction of Do pretrained models simulate humans better than instruction-tuned assistants?. The excerpt does not say which model family underlies PALATE's behavior policy, so the two notes agree on the binding signal and are silent on the substrate. On the scoring side, Can breaking persona fidelity into parts improve how we judge it? also refuses a single holistic number. There the split is along dimensions of a character's fidelity, and here it is along tracks and users, so they support the same suspicion of scalar scores from different sides.

The excerpt does not establish how large the track or user disagreements are, what the three tracks measure, which 16 candidates were compared, or how the five users were chosen and their simulators validated. With five users, "no shared best candidate" shows that a single ranking is not guaranteed. It does not estimate how much satisfaction varies across a wider population. The claim that fixed histories confound capability is stated as demonstrated in the abstract but not evidenced here. The supportable reading is modest. A one-number RPA leaderboard is an unverified summary until its rankings are checked against per-user and per-track results, and this paper reports one case where those views diverged.