Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

AI agents have become increasingly capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on relatively simple problems that finish within a few minutes and are typically evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, leading to sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a challenging terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on difficult, open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and tens of minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving.

Introduction. AI agents built on large language models (LLMs) are rapidly improving at autonomous decision making and tool use [2, 6]. Recent work has demonstrated impressive performance on short, well-scoped tasks such as fixing a code repository issue, completing a coding ticket, or issuing a handful of shell commands [22]. However, they still cover only a narrow slice of the long-horizon, domain-specific, and practically important workflows that human experts care about in practice [22, 47, 52]. Nowadays, many real workflows are long-horizon: they require agents to execute hundreds of steps, maintain and update plans over tens of minutes to hours, and manage evolving long-context state [42]. Examples include reproducing results from published research papers [36], installing environments from a github repo, auditing complex multimodal datasets, debugging compiler toolchains, or shepherding a multi-stage ML training pipeline.

Discussion / Conclusion. We introduced Long-Horizon-Terminal-Bench, a benchmark of 46 containerized terminal tasks spanning nine domains and designed to stress long-horizon execution. Unlike outcome-only benchmarks, Long-Horizon- Terminal-Bench uses deterministic, environment-grounded subtasks to provide dense partial-credit signals, allowing evaluation to capture not only whether an agent finishes a task, but also how far it progresses. Across 17 frontier models under a shared terminus-2 agent, tasks in Long-Horizon-Terminal-Bench require substantial effort, averaging 239 episodes, 9.8M tokens, $10.8 in API cost, and 88.9 minutes per run. Yet the benchmark remains far from saturated: the strongest model, Grok 4.5, achieves only a 28.3% pass rate at R ≥0.95, while the mean pass rate across models is just 6.4%. Dense rewards further reveal that many runs make meaningful but incomplete progress, exposing failure patterns that binary pass/fail would miss.