When testing an AI, should you also test how you're talking to it — giving orders versus brainstorming together?
Why do evaluation scenarios need to reproduce interaction mode dimensions?
This explores why tests of AI systems need to recreate the different ways people actually work with AI (giving orders, collaborating, exploring, getting help) instead of scoring the model on single, isolated answers.
This explores why AI evaluations should copy the real ways people interact with AI, not just test single answers. The short version is that how a person is engaging with the system changes what counts as success. One framework sorts human-AI interaction into five modes: collaborative, directive, assistive, cooperative and explorative. The modes differ in who starts the exchange and whether the human and model work on the same thing together or on separate pieces How should we categorize different types of human-AI interaction?. A model that does well when you give it a precise order can still fail as an exploration partner, where the job is to open up options. A benchmark built around only one mode quietly tests only that mode. The authors present the taxonomy as a guide for building 'ecologically valid' scenarios, meaning tests that look like the situations where the system will actually be used.
The second reason is that the evidence itself changes once interaction becomes the thing being judged. Agent evaluation is moving from scoring final answers to scoring the whole trajectory: process quality, recovery from mistakes, coordination and robustness How should we evaluate agent behavior beyond final answers?. A related argument is that interactive evaluation should be designed as a paradigm, with clear protocols and reporting standards, rather than collected as a pile of new benchmarks. Without that, results from different setups can't be compared or reproduced Should interactive evaluation be designed as a unified paradigm?. Interaction mode is one of the things those protocols would need to pin down. If one benchmark quietly assumes a directive user and another assumes a collaborative one, their scores measure different things.
Design research reaches the same point from the builder's side. A review of 291 papers found six recurring dimensions of human-GenAI systems: purpose, feedback, user control, engagement level, domain and evaluation strategy. It lists evaluation as a design dimension in its own right, not something added at the end How should designers structure human-GenAI interaction systems?. The IAI model goes further. It shows that what the model actually receives is not just the prompt but an 'augmented instruction' that also carries the interface's constraints, and it finds twelve recurring interaction patterns across 66 tools How do interaction signals reshape what generative AI actually receives?. The less obvious implication is that if you test a model with plain prompts, you are testing a different input from the one users' tools actually send.
The practical difficulty is that reproducing interaction modes at scale usually means simulating the users. Simulators can be made realistic by conditioning them on hidden variables such as user profile and per-turn intent Can controlled latent variables make LLM user simulators realistic?. One persona population can also be reused across surveys, chatbots and other interfaces through adapters Can one persona population evaluate different application types?. But simulated users drift away from their personas over long conversations, and reinforcement learning aimed at consistency cut that drift by more than half Can training user simulators reduce persona drift in dialogue?. This matters because a simulated 'explorative' user who slides into giving orders mid-conversation has changed which mode is being tested without anyone noticing.
There is also a human-perception angle. People judge dialogue agents on perceived competence, human-likeness and communicative flexibility How do users mentally model dialogue agent partners?. Those judgments probably carry different weight in different modes: flexibility matters more when exploring together than when issuing a command. That last point is an inference. The corpus doesn't test mode-by-mode weighting directly, and it is thin on studies showing the same model ranking differently across modes. Those would be the direct proof, so treat the case above as a strong argument built from adjacent findings rather than a settled result.
Sources 9 notes
Research identifies five modes—collaborative, directive, assistive, cooperative, and explorative—organized by who initiates interaction and whether humans and models work jointly or separately. This taxonomy provides a framework for building ecologically valid evaluation scenarios.
Evaluation of agentic systems shifts evidence from final responses to full interaction sequences, and scoring procedure from correctness alone to process quality, recoverability, coordination, and robustness. This pattern appears across multiple agent benchmarks as a coherent design move.
Interactive evaluation should be treated as a principled paradigm with explicit protocols and reporting standards, not adopted piecemeal as benchmarks. The fragmentation plaguing current interactive benchmarks mirrors early evaluation culture; formalizing the paradigm—expanding evidence from final responses to trajectories while standardizing how to score process quality and robustness—makes results interpretable and reproducible.
A systematic review of 291 papers identified six recurring dimensions: purposes for using GenAI, model feedback to users, user control over models, engagement levels, application domains, and evaluation strategies. Three authors collaboratively coded and synthesized these through iterative consensus.
The IAI model formalizes human-GenAI interfaces as a six-entity graph where Augmented Instruction—machine-readable encoding of prompts plus interaction constraints—is what GenAI actually receives. Applied to 66 tools, it distills twelve recurring, composable interaction paradigms and enables design comparison where prior taxonomies could not.
Show all 9 sources
RecLLM demonstrates that conditioning an LLM simulator on session-level (user profile) and turn-level (user intent) latent variables produces synthetic conversations measurable as realistic via crowdsource discrimination, discriminator models, and classifier-ensemble distribution matching.
PersonaEval demonstrates that simulated users from existing persona datasets can evaluate multiple application formats through plug-and-play interface adapters, enabling repeatable and scalable evaluation without rebuilding personas per task.
By inverting standard RL setups to train user simulators for consistency using three complementary metrics (prompt-to-line, line-to-line, Q&A consistency) as reward signals, persona drift decreases by over 55%. This approach captures distinct failure types: local drift within turns, global drift across conversations, and factual contradictions.
The Partner Modelling Questionnaire reveals that perceived competence dominates user impressions (49% of variance), followed by human-likeness (32%) and communicative flexibility (19%). This three-factor structure reflects how people evaluate dialogue partners against both functional and social standards.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Towards interactive evaluations for interaction harms in human-AI systems
- Evaluating Generative AI in the Lab: Methodological Challenges and Guidelines
- Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz
- Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration
- An HCI-Centric Survey and Taxonomy of Human-Generative-AI Interactions
- PersonaEval: Persona-Based User Simulation for Evaluating Interactive Applications
- Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning
- Interactive Evaluation Requires a Design Science