Does longer interaction horizon require fundamentally different evaluation approaches?
This explores whether evaluating AI over longer, multi-turn interactions (agents taking many steps, ongoing conversations, sustained relationships) demands genuinely new evaluation methods — or whether it's the same evaluation we already do, just stretched out.
This question asks whether stretching the interaction horizon — from single answers to long agent rollouts, multi-turn dialogue, and relationships that unfold over weeks — forces us to rethink how we measure AI, or whether existing methods just scale up. The corpus answer is split in a productive way: the *problems* don't change, but the *evidence you have to collect* does, and that's where the real shift lives.
The most direct take is deflationary. Moving to interactive, trajectory-level evaluation doesn't dissolve the hard problems of measurement — comparability, reproducibility, mapping evidence to a judgment — it relocates them into a higher-dimensional space where they're actually harder to pin down Do interactive evaluations actually solve the benchmark comparison problem?. So in one sense, no: you're chasing the same ghosts. But that's exactly why the corpus argues interactive evaluation has to be *designed* as a deliberate paradigm with explicit protocols and reporting standards, rather than adopted piecemeal as the next batch of benchmarks — otherwise the field fragments into incomparable one-off setups Should interactive evaluation be designed as a unified paradigm?.
Where a genuinely *different* approach becomes unavoidable is the temporal dimension. Several notes converge on a warning: short-horizon measurement systematically misleads about long-horizon behavior. Chatbot relationship studies show that the social processes driving early engagement decay predictably as novelty wears off, so single-session findings simply cannot be extrapolated to medium- or long-term use Do chatbot relationships lose their appeal as novelty wears off?. The failure cuts the other way too — preference optimization that looks helpful turn-by-turn quietly erodes the grounding acts (clarifying questions, understanding checks) that multi-turn reliability depends on, so a model can score well per-response and fail silently over a full conversation Does preference optimization harm conversational understanding?. Both say the same thing: evaluate at the wrong horizon and you measure the opposite of what matters.
The more surprising thread is that longer horizons don't just need *more* evaluation — they unlock *new units* of measurement that don't exist at the single-turn level. The geometry of a conversation — how it unfolds, independent of content — predicts satisfaction nearly as well as full-text analysis, capturing quality that text classifiers miss entirely Can conversation shape predict whether it will work?. Therapy transcripts can be scored turn-by-turn for working alliance, surfacing trajectory-level patterns like persistent patient-therapist misalignment in suicidality that no single turn would reveal Can we measure therapist-patient alliance from dialogue turns in real time?. And on the capability side, interaction scaling turns out to be an entirely separate axis from reasoning depth — letting an agent take more environment steps enables exploration and backtracking that no amount of per-step reasoning achieves, which means you can't evaluate a long-horizon agent with metrics built for a single smart answer Does agent interaction time scale separately from reasoning depth?.
So the honest synthesis: the horizon doesn't change what good evaluation *is*, but it changes what counts as evidence, when you have to collect it, and which failure modes only appear over time. The trap the corpus keeps flagging is borrowing a short-horizon proxy — length as difficulty Does longer reasoning actually mean harder problems?, single-session appeal, per-turn helpfulness — and assuming it survives the stretch. It doesn't. That's the thing worth knowing: long-horizon evaluation isn't single-turn evaluation scaled up, it's a different measurement problem wearing the same vocabulary.
Sources 8 notes
Interactive evaluation relocates core problems—comparability, reproducibility, evidence-to-judgment mapping—into higher-dimensional space rather than solving them. The field needs design protocols and shared standards, not format adoption, to make trajectory scoring interpretable.
Interactive evaluation should be treated as a principled paradigm with explicit protocols and reporting standards, not adopted as disconnected benchmarks. The distinction matters: designing interactive evaluation as a unified system prevents fragmentation and incomparability, while expanding what counts as evidence beyond final responses.
Longitudinal studies with Mitsuku show that social processes driving relationship formation decline as novelty wears off. Single-session study findings cannot be reliably extrapolated to medium- or long-term chatbot design.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
A structure-only model analyzing conversation trajectory achieved 68% accuracy predicting satisfaction, nearly matching full-text LLM analysis at 70%. Combined structural and textual features reached 80%, showing that how conversations unfold geometrically captures interaction quality text-based classifiers miss.
COMPASS maps dialogue turns onto WAI embeddings to produce 36-dimensional alliance scores per turn. Anxiety and depression show convergence in alliance metrics over time, while suicidality shows persistent misalignment between patient and therapist.
Test-time interaction—increasing environment steps—enables exploration, backtracking, and replanning that per-step reasoning cannot achieve. Curriculum-based RL on rollout length produces SOTA web agents, showing interaction scaling dominates on tasks with partial observability.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.