When an AI plays a therapy patient, how do you check it's staying in character from start to finish?
What psychological instruments best measure persona consistency in clinical simulation dialogue?
This explores which established psychological questionnaires or inventories could show whether a simulated patient or clinician stays in character across a therapy-style conversation. The corpus has no head-to-head comparison of such instruments, and none is tested on clinical simulation, so what follows is the closest adjacent material.
This explores which established psychological questionnaires or inventories could show whether a simulated patient or clinician stays in character across a therapy-style conversation. The corpus has no head-to-head comparison of such instruments, and none is tested on clinical simulation. What it offers instead is adjacent evidence about how consistency gets measured, and about how measurements of it mislead.
The closest thing to a consistency yardstick isn't a psychological instrument. One approach trains user simulators against three separate checks: does each line fit the persona prompt, does it fit the previous lines, and does the simulator answer questions about itself the same way each time. Using those checks as a reward signal cut persona drift by more than 55% Can training user simulators reduce persona drift in dialogue?. The checks catch different failures: drift within a turn, drift across a whole conversation, and outright factual contradiction. That split fits clinical simulation well, because a simulated patient who reports sleeping fine in turn 4 and insomnia in turn 30 has failed differently from one who drifts into a generic, upbeat tone. A related idea gives the agent an imaginary listener and asks whether an utterance would distinguish its persona from a distractor. The paper uses this to reduce contradictions during generation without extra labels Can imaginary listeners reduce dialogue agent contradictions?. Turning that test into a scoring method for transcripts would be my inference, not something the paper does.
The library's two psychology-flavored instruments measure something next door. COMPASS maps therapy turns onto Working Alliance Inventory embeddings, giving a 36-dimensional alliance score for every turn. In real transcripts, anxiety and depression cases converge over time, while suicidality shows persistent misalignment between patient and therapist Can we measure therapist-patient alliance from dialogue turns in real time?. That measures the relationship, not the persona. It does suggest a test: a simulated patient with a given profile should trace the alliance trajectory that profile produces in real sessions. Nothing in the corpus tries this. The Partner Modelling Questionnaire is a genuine psychometric tool, but it captures how users perceive a dialogue agent: competence (49% of variance), human-likeness (32%), and communicative flexibility (19%) How do users mentally model dialogue agent partners?. It measures perceived consistency, not actual consistency.
The corpus also warns against trusting any single score. High persona-adherence scores often come from copying the character description while ignoring what the other speaker said, so persona fidelity and conversational coherence have to be measured together Do persona consistency metrics actually measure dialogue quality?. A layered, scripted persona produced more human-like dialogue consistency but weaker emotional expression Can layered persona architecture sustain coherent character behavior?. That trade-off matters for clinical simulation, where emotional expression is the thing being simulated. Realism itself has been checked with crowdsourced discrimination, trained discriminator models, and classifier-ensemble distribution matching Can controlled latent variables make LLM user simulators realistic?.
One more angle is stress-testing rather than scoring. Post-training appears to install personas that hold under adversarial pressure, while prompt-induced role-play tends to collapse Are RLHF personas performed characters or realized dispositions? Are LLM personas realized or merely simulated through training?. For a clinical simulator, that suggests measuring consistency as whether the patient stays in character when pushed with leading questions or pushback. A single-pass questionnaire score wouldn't capture that. Overall, the corpus supports a layered approach: turn-level and contradiction checks, an alliance-style clinical signal, and pressure tests. It does not support naming one best instrument.
Sources 9 notes
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.
Endowing dialogue agents with an imaginary listener via Rational Speech Acts reduces persona contradiction at inference time without NLI labels or extra training. The agent simulates whether utterances would distinguish its persona from a distractor, suppressing generic or contradictory responses.
COMPASS maps dialogue turns onto WAI embeddings to produce 36-dimensional alliance scores per turn. Anxiety and depression show convergence in alliance metrics over time, while suicidality shows persistent misalignment between patient and therapist.
The Partner Modelling Questionnaire reveals that perceived competence dominates user impressions (49% of variance), followed by human-likeness (32%) and communicative flexibility (19%). This three-factor structure reflects how people evaluate dialogue partners against both functional and social standards.
High persona adherence scores often come from copying character descriptions while ignoring query relevance. MUDI jointly optimizes both by using discourse relations and graph-based coherence modeling alongside persona fidelity, showing that persona and context must be optimized together, not separately.
Show all 9 sources
Deep Persona's three-layer architecture, which restricts the model to reactive response within a structured script, shows dialogue more closely aligned with human conversation patterns and achieves high pragmatic fluency, though with limitations in emotional expression.
RecLLM demonstrates that conditioning an LLM simulator on session-level (user profile) and turn-level (user intent) latent variables produces synthetic conversations measurable as realistic via crowdsource discrimination, discriminator models, and classifier-ensemble distribution matching.
Post-training installs stable dispositional profiles that persist under adversarial pressure, marking them as realized rather than performed. The stickiness of trained personas across conversations distinguishes them from prompt-induced role-play that collapses under jailbreaks.
Post-training installs robust personas that resist adversarial pressure and persist as substrate-level dispositions, distinguishing realization from pretense. This quasi-realizationist account preserves explanatory power while treating LLMs as possessing genuine quasi-beliefs and quasi-desires.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- Do LLMs Understand Personality? Rethinking Persona Fidelity Evaluation through Structured Behavioral Inference
- Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations
- The Assistant Axis: Situating and Stabilizing the Default Persona of Language Models
- The Illusion of Debiasing: Persona Steering Redistributes Rather Than Reduces Bias in LLMs
- Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness
- From Persona to Person: Enhancing the Naturalness with Multiple Discourse Relations Graph Learning in Personalized Dialogue Generation
- Building Persona Consistent Dialogue Agents with Offline Reinforcement Learning