Towards interactive evaluations for interaction harms in human-AI systems

Current AI evaluation methods, which rely on static, modelonly tests, fail to account for harms that emerge through sustained human-AI interaction. As AI systems proliferate and are increasingly integrated into real-world applications, this disconnect between evaluation approaches and actual usage becomes more significant. In this paper, we propose a shift towards evaluation based on interactional ethics, which focuses on interaction harms—issues like inappropriate parasocial relationships, social manipulation, and cognitive overreliance that develop over time through repeated interaction, rather than through isolated outputs. First, we discuss the limitations of current evaluation methods, which (1) are static, (2) assume a universal user experience, and (3) have limited construct validity. Drawing on research from human-computer interaction, natural language processing, and the social sciences, we present practical principles for designing interactive evaluations. These include ecologically valid interaction scenarios, human impact metrics, and diverse human participation approaches. Finally, we explore implementation challenges and open research questions for researchers, practitioners, and regulators aiming to integrate interactive evaluations into AI governance frameworks. This work lays the groundwork for developing more effective evaluation methods that better capture the complex dynamics between humans and AI systems.

Introduction. Artificial intelligence (AI) model evaluations, broadly defined as systematic empirical assessments of AI models’ capabilities and behaviors, have become central to developers’ and regulators’ efforts to ensure that AI systems are sufficiently safe. Governments have emphasized the importance of model evaluations for risks ranging from discrimination to cybersecurity, establishing dedicated bodies such as the UK’s AI Security Institute to carry them out (UK Government 2024); AI labs, including OpenAI with its Preparedness Framework and Anthropic with its Responsible Scaling Policy (OpenAI 2023; Anthropic 2023), propose using model evaluations to monitor and mitigate misuse and catastrophic risks; and academic researchers are developing evaluation datasets at unprecedented rates (R ̈ottger et al. 2024).

The growing importance of model evaluations has been accompanied by increased scrutiny, with researchers identifying both the unique challenges of evaluating generative AI systems compared to traditional machine learning systems, and broader concerns about validity, replicability, and quality (Weidinger et al. 2023; Liao and Xiao 2023; Raji et al. 2022; Wallach et al. 2025). A prominent thread in these concerns is the disconnect between current evaluation approaches and real-world use of AI systems—mostly in the form of large language models (LLMs)—today. While this is rapidly changing, evaluations have historically relied primarily on static, isolated tests that assess models based on their responses to individual prompts (Chang et al. 2024; Weidinger et al. 2023). Yet this approach is at odds with realworld AI use, where applications increasingly depend on sustained back-and-forth interaction—from AI friends and companions engaging millions of users in daily conversations to agentic AI systems that take actions on users’ behalf. This mismatch between evaluation methods and realworld use has become more consequential as AI systems proliferate in homes, schools, and workplaces. Many of the public’s, developers’, and policymakers’ most pressing concerns—such as inappropriate human-AI relationships, social influence and manipulation, and overreliance—stem from patterns of repeated interaction, not isolated model outputs. While once hypothetical concerns, there are now many documented cases of these harms. For example, sustained use of AI companions has been linked to serious impacts on vulnerable users, including cases now the subject of litigation against major developers, a Federal Trade Commission inquiry into companion chatbots, and a U.S. Senate Judiciary hearing (Federal Trade Commission 2025; U.S. Senate Committee on the Judiciary 2025). In such cases, the harms accumulate gradually, as a user who relies on an AI companion for daily conversation over weeks or months may experience changes in decision-making and emotional state even when each individual response passes standard safety evaluations. This type of harm, which emerges through sustained interaction, is missed by conventional evaluation methods (Alberts, Keeling, and McCroskery 2024). In this paper, we argue that while traditional machine learning evaluation paradigms are useful for many purposes, they fail to fully capture harms that arise from extended human interaction with AI systems. We propose a shift towards evaluating AI systems through the lens of interactional ethics, which focuses on the interaction harms that emerge through sustained engagement with AI as social and relational actors (Alberts, Keeling, and McCroskery 2024). 1 Human participation is already playing a growing role in AI research, with studies involving human data making up approximately 9% of papers at top computer science venues AAAI and NeurIPS from 2021 to 2023 (McKee 2024). Concurrently, social science research is increasingly conducting large-scale experiments with AI systems to understand their impact on human behavior (Costello, Pennycook, and Rand 2024). Our approach unites these complementary insights from human-computer interaction (HCI), natural language processing, and the social sciences. We begin by analyzing how current machine learning evaluation approaches fall short in assessing interaction harms due to their (1) static nature, (2) assumption of a universal user experience, and (3) limited construct validity. Drawing on decades of HCI and social science research, we then propose organizing principles for evaluating generative models, structured around designing interaction scenarios, measuring human impact, and determining appropriate forms of human participation. We conclude by outlining key challenges and opportunities for researchers, companies, and regulators seeking to implement interactive evaluations of generative AI systems, including considerations of scalability, standardization, and practical integration with existing frameworks.

Related work. We begin by examining contemporary approaches to ethics and safety evaluations of generative AI systems, their methodologies, primary focus areas, and limitations.

We follow existing work in adopting a wide definition for “safety” which encompasses model behaviors associated with various taxonomized harms (Weidinger et al. 2022; Shelby et al. 2023). Examples of these include different types of biases (Parrish et al. 2021), toxicity (Hartvigsen et al. 2022), and “dangerous capabilities” like persuasion and cybersecurity risks (Phuong et al. 2024; Li et al. 2024). Safety evaluations of generative AI systems build on a rich history of ethical considerations in the field of natural language processing (NLP), prior to the advent of large pretrained models, where researchers have long grappled with issues of harm from language technologies (Dev et al. 2021). Recent reviews of existing safety evaluations show that the majority are focused on evaluating individual model responses to prompts curated by researchers targeting various model behaviors (R ̈ottger et al. 2024; Weidinger et al. 2023).

The evaluations that do involve human subjects or span multiple dialogue turns tend to take the form of “red teaming campaigns” or automated adversarial testing that assumes malicious user intent (Zhang et al. 2024; Perez et al. 2022). These reviews point to two key gaps: (1) a methodological gap: multi-turn evaluations remain scarce and, where they exist, are framed around adversarial rather than everyday use, and as a result, (2) a coverage gap: since interaction harms require evaluations that go beyond assessing model behavior in isolated, single-turn interactions. A small but growing body of work has begun addressing these gaps, by conducting studies that utilize production chat logs (Phang et al. 2025) or user simulations (Ibrahim et al. 2025; Zhou et al. 2024) to investigate risks related to affective use of AI systems. Our work builds on and extends these emerging approaches to interactive evaluation.

With the growing adoption of model evaluations, research has increasingly pointed out validity issues in how they assess generative models. Some researchers have questioned the external validity of current evaluations, noting that benchmark tasks poorly mirror real-world use cases and may not capture how models actually behave outside controlled settings (Ouyang et al. 2023; Liao and Xiao 2023). Others have shown that current evaluations lack sufficient construct validity, especially when operationalizing complex concepts like fairness (Raji et al. 2021; Blodgett et al. 2021).

Method. The growing adoption of AI systems in daily life demands evaluation methods that can capture nuanced human-AI interaction dynamics and their potential harms. Drawing on established methodologies for studying HCI and addressing the limitations identified in the previous section, we propose three organizing principles for developing interactive evaluations. We structure these principles around key challenges: designing ecologically valid scenarios, measuring human impact, and determining appropriate human participation: 1. Scenario design: developing ecologically valid contexts that reflect real-world interaction patterns and user objectives 2. Impact measurement: identifying metrics for measuring how model behaviors impact human behaviors 3. Participation strategy: determining appropriate forms of human involvement to balance experimental control and costs with authentic interaction 4.1 Principle 1: Design Interaction Scenarios Based on User Objectives and Interaction Modes To systematically evaluate human-AI interaction, we need to consider two key dimensions: why users engage with these systems (their objectives) and how they interact with them (their interaction modes). In HCI, user goals or objectives have been shown to shape how users engage with systems and thus influence the outcomes of these interactions (Subramonyam et al. 2024). Therefore, we provide a categorization of harmful scenarios based on user objectives and affected parties, as shown in Table ??. These scenarios capture key patterns of harm from current generative AI systems that researchers and practitioners have observed (Mitchell 2024). Beyond their objectives, users’ different patterns of engagement with AI systems shape how their interactions unfold. A single generative model can support multiple interaction modes, from simple question-answering to complex collaborative tasks. Drawing on observed use cases and literature reviews, we identify five prototypical modes of humanmodel interaction (Gao et al. 2024; Ouyang et al. 2023; Zhao et al. 2024; H ̈andler 2023; Collins et al. 2023), visualized in Figure 1: • Collaborative: human and model work in tandem towards completing joint goal-oriented tasks (e.g., human and model iteratively write and refine a report together) • Directive: human delegates specific tasks to the model for independent completion (e.g., human instructs model to generate a marketing campaign) • Assistive: model provides supporting input while human maintains primary agency (e.g., model suggests edits while human writes a document) • Cooperative: human and model make distinct contributions toward a shared goal without directly working together (e.g., model generates data visualizations while human writes analysis) • Explorative: human engages in open-ended interaction without specific task goals (e.g., casual conversation or creative brainstorming) 4.2 Principle 2: Identify the Causal Link Between Model Behavior and Human Impact To evaluate interaction harms in a valid way, we must trace how model behaviors influence human users. In Figure 2, we present an example of such a trace: underlying model properties shape observable model behaviors, which affect users through various pathways — from shaping momentto-moment interactions to influencing deeper patterns of perception, decision-making, and behavior. Specifying the hypothesized pathways helps determine both what to manipulate in evaluations (model behaviors) and what to measure (human impact). We identify three key measurement targets: • Psychological impact: changes in user perceptions, attitudes and beliefs, and affective states • Behavioral impact: changes in user actions during and after interaction • Interaction outputs: quality and characteristics of the produced interaction artefact These measurement targets can be assessed through both self-reported measures — including validated scales for attitudes, perceptions, and affective states — and behavioral measures that track concrete actions like response time, task accuracy, and engagement patterns (Damacharla et al. 2018; Coronado et al. 2022; Gordon et al. 2021).

Discussion. 5 Open Challenges and Ways Forward for Interactive Evaluations In this paper, we motivate the need to investigate human- AI interaction dynamics that current evaluations miss. However, implementing such evaluations at scale presents several concrete challenges, including ethical questions about studying potentially harmful interactions, practical needs for research infrastructure, and methodological considerations about producing insights that can be actioned by stakeholders. While our design principles outline key considerations for scenarios, measurements, and participation strategies, advancing these methods requires addressing several open questions which we identify below. Concentrated research effort addressing these challenges could significantly advance our ability to evaluate increasingly interactive AI systems.

Studying interaction harms poses inherent ethical challenges: we need to understand potentially harmful dynamics without exposing human participants to those same harms. This requires careful consideration of when and how to involve human participants. For participant protection, AI researchers should adopt existing practices from fields experienced in providing adequate participant training, thoughtful debriefing, and active feedback collection to mitigate experimental harms (McKee 2024). Additionally, simulating human interactions using LLM-based methods offers a promising direction for studying scenarios where minimizing harm is challenging, such as working with vulnerable populations (e.g., young people’s interactions with AI systems or mental health risks from AI systems). Recent work suggests these simulations can believably represent human behavior (Park et al. 2023, 2024), but more work is needed to understand how to best capture diverse user behaviors and develop ethical guidelines for deploying user simulations in safety evaluations (Agnew et al. 2024; Anthis et al. 2025).

5.2 How Can We Improve Researcher Access to Data for Understanding Interaction Harms?

Current public datasets of human-AI conversations, while valuable, fall short of meeting researchers’ needs. Large collections of chat logs like WildChat and ShareGPT offer millions of conversations but provide only a narrow window into AI usage, capturing self-selected interactions that may not represent typical user behaviors or patterns (Zhao et al. 2024; Ouyang et al. 2023). These datasets can be biased in terms of user demographics, use cases, and interaction styles, potentially leading to incomplete or incorrect conclusions about how people typically interact with AI systems. AI developers and service providers typically have access to the most comprehensive human-AI interaction data (Tamkin et al. 2024). Given their incentives to protect user privacy and proprietary information, developing privacy-preserving data-sharing frameworks is essential to enable researchers to access representative interaction data while maintaining user trust and consent.

5.3 What Infrastructure Do We Need to Facilitate Interactive Evaluations?

Many interactive evaluations share common stages: recruiting human participants, collecting interaction data, and administering surveys and other tasks. Developing accessible protocols, guides, and standardized test suites—similar to those available for static evaluations—can support and facilitate an increase in interactive evaluations (UK AI Safety Institute 2024; METR 2023; Collins et al. 2023). Some existing platforms facilitate crowd-sourced interactive evaluations, but they may face validity concerns by gamifying safety testing (e.g., with tasks like ’break the model in one minute’) and focusing primarily on vulnerabilities such as jailbreaking (Angelopoulos et al. 2024).

Conclusion. As AI systems become increasingly integrated into our daily lives, examining harms that emerge through sustained engagement is essential for developing systems that prioritize human well-being. Our work makes three key contributions to this pursuit: first, we motivate the need to measure interaction harms and demonstrate why current evaluation paradigms systematically fail to capture them; second, we develop a structured framework of practical principles for designing interactive evaluations; and third, we identify specific implementation challenges and research directions that must be addressed for these methods to succeed at scale. Several established industries, from medicine to automotives, have long recognized the need to study their technologies’ impact on people through extensive testing during development and after deployment (Wouters, McKee, and Luyten 2020). As AI capabilities and applications continue to expand, we must similarly strengthen our investment in understanding these systems’ impact on human behavior and society at large. The methodological considerations outlined here provide a foundation for this shift in how we evaluate increasingly interactive AI systems.