Do static AI evaluations miss harms that build over time?

The paper argues that AI model evaluations have "relied primarily on static, isolated tests that assess models based on their responses to individual prompts," a method mismatched with real-world use "where applications increasingly depend on sustained back-and-forth interaction." It names the harms this disconnect misses as "interaction harms—issues like inappropriate parasocial relationships, social manipulation, and cognitive overreliance that develop over time through repeated interaction, rather than through isolated outputs." As evidence of stakes, it points to AI companion use now the subject of litigation against major developers, a Federal Trade Commission inquiry into companion chatbots, and a U.S. Senate Judiciary hearing. The core example: "a user who relies on an AI companion for daily conversation over weeks or months may experience changes in decision-making and emotional state even when each individual response passes standard safety evaluations."

The mechanism is a three-part validity critique of current evaluation practice: it is static (single prompt-response pairs rather than trajectories), it assumes a universal user experience, and it has limited construct validity for complex concepts. The paper proposes replacing this with evaluation grounded in "interactional ethics," organized around three principles — designing ecologically valid interaction scenarios tied to user objectives and interaction modes, measuring human impact (psychological, behavioral, and interaction-output changes) rather than only output quality, and choosing participation strategies that balance experimental control against authentic interaction.

This sharpens a pattern already visible elsewhere in the library: Do chatbot safety measures accidentally increase emotional entanglement risks? names relational harms that emerge the same way — through accumulation rather than any single exchange — and this paper supplies the evaluation-design argument for why static tests would miss exactly that kind of harm. It runs parallel to Should interactive evaluation be designed as a unified paradigm?, which makes the general case that interactive evaluation needs principled design rather than ad hoc benchmark adoption; this paper is the harms-specific instance of that same argument, aimed at psychological and relational risk rather than agent task performance. It also gives the evaluation-methodology counterpart to the empirical finding in Does how much time people spend with chatbots drive worse outcomes?, which demonstrates in a trial exactly the kind of accumulated, usage-driven harm this paper says static single-turn tests cannot detect.

The excerpt is a position and framework paper, not an empirical study: it does not report new measurements, specific metrics, or a tested evaluation protocol, and it leaves open how to recruit and protect participants, how researchers gain access to representative interaction data, and what shared infrastructure interactive evaluations would need. The implication it supports is narrower than interactive evaluation works — only that static, single-turn evaluation is structurally unable to capture harms that are defined by their accumulation over repeated interaction, which argues for supplementing, not replacing, existing model evaluations.