Is an AI role-player just 'good' or 'bad' — or does its quality depend on who it's talking to?

This explores whether role-play evaluation should score an agent's general ability or how well a specific agent and a specific user work together. The corpus leans against pure ability scores. Standard benchmarks hand the agent a borrowed dialogue history and grade its next line against a fixed rubric. Does fixed dialogue history bias role-play agent evaluation? argues this mixes up what the agent can do with the context it was given, and hides differences between users. PALATE instead pairs each agent with a simulator trained on a real user's history. It finds that no single agent wins across all users and evaluation tracks. If the ranking flips depending on who is on the other side, ability is not a property of the agent alone.

Research on how people judge dialogue partners points the same way. How do users mentally model dialogue agent partners? finds that impressions of an agent split into perceived competence (49% of the variance), human-likeness (32%) and communicative flexibility (19%). Competence dominates, but about half of what users react to is something else. All three are judged from one person's seat, so a single ability number squashes a three-part judgment. The note doesn't measure pair fit itself, but the human-likeness and flexibility factors are where two users would most plausibly disagree about the same agent.

Measuring fit has a cost, because the user in the pair is now a simulation and the score is only as good as the simulator. Can training user simulators reduce persona drift in dialogue? shows that simulated users drift from their persona within a turn, across a conversation, and into factual contradictions. Training against consistency rewards cut that drift by over 55%. A drifting simulator would make a fit score measure the simulator's noise. Can AI personas reliably replicate human experiment results? adds a warning about size: persona simulations reproduced 84 of 111 published main effects, tracking how strong the original evidence was, but weak effects gave both false positives and false negatives. Small differences in fit between agents are the kind of signal simulation handles worst. The same caution applies to automated graders. Can agents evaluate AI outputs more reliably than language models? cut judge shift from 31% to 0.27%, yet one module cascaded errors through the whole pipeline.

The corpus suggests reporting both. Can one persona population evaluate different application types? shows one persona population can be reused across different applications through interface adapters, and Can personas extracted from documents generalize across evaluation tasks? shows document-grounded personas transferring across tasks. With a fixed, reusable user population you can vary only the agent. The average across users is the closest thing to ability, and the spread between users is the fit. What the notes don't show is whether a fit-based ranking predicts real user satisfaction better than an ability ranking, so that test is still missing.