AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities

As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an opensource, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.

Introduction. The paradigm of Large Language Models (LLMs) is rapidly shifting from instruction-following text generators to the foundation of LLM-based agents capable of complex reasoning, planning, and tool interaction in dynamic environments [17, 38]. As LLM-based agents continue to advance in capability, corresponding evaluation methodologies must be developed in tandem to ensure rigorous assessment of these increasingly multi-faceted systems. However, the current agent evaluation landscape is highly fragmented, suffering from a severe infrastructural deficit. While numerous specialized benchmarks have emerged to assess specific capabilities—such as tool invocation [46] and deep research [22]—they operate as isolated evaluation suites. This forces researchers to repeatedly configure heterogeneous execution environments, data formats, and scoring protocols. Such redundant engineering not only hinders efficiency but also compromises reproducibility due to inconsistent baseline implementations.

Discussion / Conclusion. In this paper, we introduced AgentCompass, an open-source and highly extensible infrastructure designed to systematize the evaluation of LLM-based agents. By decoupling the evaluation pipeline into independent Model, Benchmark, Harness, and Environment components, it eliminates redundant engineering and ensures rigorous reproducibility. Supported by a fault-tolerant asynchronous runtime and granular trajectory analysis, AgentCompass goes beyond traditional scalar scores to help researchers diagnose complex behaviors such as reward-hacking. With native integration of over 20 diverse benchmarks, AgentCompass provides a robust and scalable infrastructure to accelerate the next generation of agent research.