Do short benchmarks predict how models perform over long workflows?
Standard LLM benchmarks measure single-turn performance, but real workflows involve sustained delegation across many turns. The question explores whether top benchmark performers maintain accuracy through longer interaction chains.
Most LLM benchmarks evaluate single-turn or short-multi-turn interaction. DELEGATE-52 extends evaluation to 50-round-trip relays and finds that short-interaction performance is not predictive of how the same model behaves under sustained delegation. Models that perform comparably on a single edit can diverge dramatically by relay 25.
This is a methodological finding, not a model finding. The standard practice — pick the top scorer on benchmark X, deploy it in workflow Y — implicitly assumes that capability is roughly stationary across interaction lengths. The relay results show the assumption fails. Models exhibit a degradation curve, and that curve has its own shape parameters (slope, decay rate, recovery behavior under interrupted sessions) that benchmarks built for short tasks cannot expose.
The implication is that "long-horizon performance" deserves status as a distinct evaluation axis, not as a property to be inferred from single-step competence. A model with strong relay-50 retention but mediocre single-turn polish may be more useful for delegated work than the inverse. The paper argues this directly: capability research has been investing heavily in memory management while leaving the underlying long-interaction degradation profile under-measured.
For practitioners, this changes the deployment question from "which model scores highest on X" to "which model maintains accuracy through the interaction length my workflow requires." For benchmark designers, it argues for relay-style evaluations as a default rather than an add-on.
Inquiring lines that read this note 28
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What systematic failures occur in human preference-based model evaluation? How do dialogue structure and state management affect multi-turn reasoning quality? What explains the gap between LLM semantic understanding and production? How do benchmark scores relate to real-world task completion?- What deployment context determines which benchmark mode actually matters?
- How should benchmarks evaluate workflow architecture versus raw model performance?
- What is the gap between benchmark performance and real workplace task completion?
- Why do short interaction benchmarks fail to predict long horizon performance?
- Do reasoning benchmarks predict real performance in long delegated workflows?
- What real-world tasks most clearly expose gaps between benchmark performance and actual capability?
- How much do prompts and data splits shift a single benchmark score?
- Can the benchmark-performance mismatch be estimated reliably without an oracle?
- How similar must finetuning data be to a benchmark to predict performance?
- Should long horizon performance be measured as a separate evaluation axis?
- How should single-axis benchmarks account for separable capability dimensions?
- Can single benchmark scores predict whether a model is ready to deploy?
- Can a single axis benchmark ever represent deployment readiness accurately?
- Can a single benchmark score capture both progress and readiness?
- Can automated benchmarks accurately capture progress on real-world long-horizon tasks?
- Why do benchmark tasks differ from real occupational workflows in practice?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do frontier LLMs silently corrupt documents in long workflows?
DELEGATE-52 tests whether state-of-the-art language models reliably preserve document integrity across extended delegated tasks. Understanding this matters because single-step benchmarks may mask compounding failures that emerge only at workflow scale.
same paper, the underlying phenomenon
-
Are LLM and agent benchmarks really measuring different things?
Do LLM benchmarks and agent benchmarks test fundamentally different capabilities, or are they two modes of the same model? Understanding this shapes how we evaluate and develop AI systems.
same paper, complementary methodology implication
-
Do models fail worse when their own errors fill the context?
As a model's prior mistakes accumulate in context, does subsequent accuracy degrade predictably? And can scaling or architectural changes prevent this self-contamination effect?
adjacent: known long-horizon failure mode
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- LLMs Corrupt Your Documents When You Delegate
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- How Well Can AI Do Strategy? Empirical Benchmarking Using Strategy Simulations
- LLMs Get Lost In Multi-Turn Conversation
- UserBench: An Interactive Gym Environment for User-Centric Agents
- Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- AI Agents That Matter
Original note title
short-interaction LLM benchmarks do not predict long-horizon delegated-workflow performance