SYNTHESIS NOTE
Topics›Evaluations›this note

Should interactive evaluation be designed as a unified paradigm?

As AI systems increasingly act over time through tools and environments, how should we structure evaluation of these interactions? Current benchmarks are fragmented and incomparable, raising the question of whether interactive evaluation needs principled design standards rather than ad-hoc adoption.

Synthesis note · 2026-05-28 · sourced from Evaluations

AI evaluation is undergoing a structural change: models are increasingly deployed as systems that act over time through tools, environments, users, and other agents. Yet most evaluation practice still inherits response-centered assumptions — fixed inputs, isolated outputs, a judgment made from a single response. Interactive benchmarks have proliferated, but the landscape is fragmented: they disagree on what interaction artifacts they admit, how trajectories are scored, and what claims their results support. This paper's position is that interactive evaluation should be treated as a principled paradigm, not as the next family of agent benchmarks to collect.

The argument turns on a definition: evaluation is an autonomous mapping E: X → Y from admissible evidence X to judgments Y. Interactive evaluation changes both sides. The evidence X expands from final responses to interaction-generated trajectories; the procedure E must assess not just final correctness but process quality, recoverability, coordination, safety, efficiency, and robustness. From this the authors build a two-axis taxonomy (what artifacts enter; how they map to judgments), derive design principles and reporting standards, and locate where current benchmarks concentrate and what they miss.

Why it matters: the distinction between designing and adopting is the whole point. Adopting interactive benchmarks one at a time produces incomparable, non-reproducible, non-extensible scores — the same fragmentation that plagued early benchmark culture, now at the trajectory level. Treating interactive evaluation as a design science forces explicit protocols, richer trajectory measures, shared infrastructure, and reporting standards that make scores interpretable. The counterpoint the paper concedes: response-centered evaluation remains useful — it is insufficient, not wrong — so the paradigm shift is additive, expanding what counts as evidence rather than discarding the old measures.

Inquiring lines that read this note 25

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How should evaluation expand to capture deployment realities? How can conversational AI develop appropriate initiative and goal awareness? Can AI systems achieve sustainable self-improvement without external human guidance? Does polished AI presentation mislead people about accuracy? How should evaluation environments be designed as security boundaries? What causes models to fail in specialized domains versus generally? Why do AI agents struggle to generalize and improve reliably? What evaluation metrics beyond task success matter most for agents? How reliably can LLM-based persona simulations predict human behavior? How do standardized benchmarks differ from measures of real-world AI capability? Why do credentials lose value when evaluation formats retire? How does increased AI involvement reshape human agency and accountability? Why do LLMs succeed at generation but struggle with evaluation? How can evaluation systems stay reliable and resistant to AI-driven gaming? What organizational factors determine whether firms successfully adopt and integrate AI? Does generative AI shift knowledge work away from collaborative tasks? What makes decomposed prompts more reliable than monolithic instructions? How does interface design shape effectiveness in human-AI collaboration? When do people apply social norms to AI systems?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
19 direct connections · 196 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

interactive evaluation must be designed as a paradigm not adopted as the next benchmark format