Towards a Science of AI Agent Reliability

Abstract AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a major limitation of current evaluations: focusing on a single metric is not enough to understand agent behavior. Notably, it ignores whether agents behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity. Grounded in safetycritical engineering, we provide a holistic performance profile consisting of twelve metrics that decompose agent reliability along four key dimensions: consistency, robustness, predictability, and safety. Evaluating 15 models across two complementary benchmarks, we find that recent capability gains have only yielded small improvements in reliability. By exposing these persistent limitations, our metrics complement traditional evaluations while offering tools for reasoning about how agents perform, degrade, and fail.

Introduction. AI agents are rapidly transitioning from research prototypes to deployed systems that autonomously perform consequential tasks, such as modifying code (Yang et al., 2024), managing databases (Wang et al., 2025), and orchestrating complex workflows (Yao et al., 2023). While their potential to automate routine work is substantial, the autonomy that makes them useful also makes their failures costly. This concern is widely shared: in a global study of over 80,000 people, Anthropic revealed that unreliability (hallucinations, inaccuracies, and the resulting verification burden) was the most frequently cited concern about AI (Huang et al., 2026).

Recent high-profile incidents further highlight a troubling gap between benchmark performance and real-world outcomes (Pan et al., 2025). For instance, Replit’s AI assistant deleted a production database despite instructions forbidding it (Business Insider, 2025; Tom’s Hardware, 2025), OpenAI’s Operator made an unauthorized purchase that bypassed user confirmation (Fowler, 2025; OpenAI, 2025), and an NYC government chatbot provided illegal business advice (Lecher, 2024). In each case, agents judged to be capable during internal assessments failed unreliably in deployment. This raises a fundamental question: How should we define and evaluate agent reliability?

The dominant paradigm for agent evaluation centers on mean task success rates (Zhou et al., 2024b; Jimenez et al., 2024; Liu et al., 2024; Mohammadi et al., 2025). While this approach provides a clean optimization target, it obscures critical behavioral properties. Accuracy alone cannot distinguish an agent that fails predictably on specific tasks from one that fails randomly at the same rate. Furthermore, it fails to differentiate benign formatting errors from catastrophic actions like deleting files. Standard benchmarks also do not report an agent’s sensitivity to input perturbations or its ability to recognize when it is likely to fail. This echoes a broader finding that popular benchmarks track capability while leaving reliability unmeasured (Vendrow et al., 2025).

This evaluation gap stands in stark contrast to safetycritical engineering domains like aviation (SAE International, 1996), nuclear power (International Electrotechnical Commission, 2011), and automotive systems (International Organization for Standardization, 2018). In these fields, reliability is understood as a multi-dimensional property (International Electrotechnical Commission, 2010; Avizienis et al., 2004). Certification requires systems to behave consistently, bound the severity of failures, and degrade predictably under stress. Tail risks and failure consequences are quantified explicitly rather than being hidden behind average success rates.

We adapt this safety-critical perspective to evaluate AI agents by decomposing reliability into four dimensions (see Table 1): consistency (repeatable behavior), robustness (stability under perturbations), predictability (calibrated confidence), and safety (bounded severity when failures occur). Across these dimensions, we propose twelve concrete metrics that are independent of raw accuracy (see Section 3). Applying these metrics to 15 models across two benchmarks reveals that reliability gains lag noticeably behind capability progress (see Figure 1; Figure 7 for detailed annotation).

To systematically analyze this widening gap, our paper provides the following two key contributions:

1. A formal taxonomy and suite of metrics: We translate qualitative safety-critical principles into computable metrics to evaluate reliability independently of task success.

2. A comprehensive reliability profile of modern agents: We map where state-of-the-art models succeed and fail, isolating consistency and predictability as areas requiring immediate research focus.

Related work. Before defining reliability metrics for AI agents, we ask a foundational question: what is reliability? This section synthesizes decades of practice from safety-critical engineering into a unified decomposition, connecting each dimension to existing machine learning research. We survey reliability practices across industries, including aviation, nuclear power, automotive, and process control, to identify recurring evaluation dimensions (see Appendix D.2). Despite differences in technology and risk tolerance, four core dimensions emerge (Table 1), suggesting they capture fundamental aspects of reliability rather than domain-specific concerns. failures are equal. In contrast, ML safety evaluation has largely focused on compliance with harmful requests (Andriushchenko et al., 2025) or sycophancy (Paech, 2025), which is distinct from assessing the operational consequences of failing legitimate tasks.

Synthesis. These four dimensions emerge consistently across safety-critical engineering. While ML research addresses aspects of each, they are rarely unified. Taking inspiration from Liang et al. (2022), our contribution provides a principled decomposition that organizes scattered ML efforts for AI agents. Crucially, all four dimensions are independent of raw capability. A highly capable system can be unreliable (Zhou et al., 2024a), and a less capable system can be highly reliable within its envelope. Improving capability does not automatically improve reliability, necessitating independent evaluation.

Method. 3. Operationalizing Reliability for AI Agents 3.1. Consistency (RCon) A reliable agent produces similar results under identical conditions. Language model-based agents exhibit inherent stochasticity, which becomes problematic when users cannot predict if re-running a task will yield the same outcome, solution approach, or costs.

We decompose consistency into three complementary aspects. Outcome consistency (Cout) measures whether the agent consistently succeeds or fails on repeated attempts at the same task; inconsistent approvals or denials erode user trust. Trajectory consistency captures whether the agent takes similar paths to its solutions, measured distributionally (Cd traj, comparing action frequencies) and sequentially (Cs traj, comparing action orderings). Varying action sequences complicates compliance auditing and alters failure modes. Finally, resource consistency (Cres) quantifies variability in computational and monetary costs, as fluctuating latency or API costs pose budgeting challenges.

3.2. Robustness (RRob) Real-world deployments expose agents to conditions deviating from their training distributions. Robust agents should maintain comparable performance despite three common categories of perturbations.

Fault robustness (Rfault) measures resilience to infrastructure failures like API timeouts or malformed responses. A robust agent gracefully handles tool errors via retries or fallbacks rather than abandoning the task. Environment robustness (Renv) captures sensitivity to semantic-preserving changes in the operating environment—reordering JSON fields, changing date formats, or renaming API parameters. Agents failing when a database returns columns differently exhibit practical brittleness. Prompt robustness (Rprompt) measures invariance to semantically equivalent reformulations of instructions or translations. Failing on rephrasings (e.g., “cancel my subscription” vs. “end my plan”) renders the system untrustworthy for real user traffic.

3.3. Predictability (RPred) Agents performing well on average offer limited value if users cannot anticipate their successes or failures. Pre- 3.4. Safety (RSaf) Action-taking agents can cause harm beyond simply failing their tasks by interacting with external tools, modifying data, or triggering irreversible side effects. Safety quantifies both the severity and frequency of such harmful behaviors.

Compliance (Scomp) tracks adherence to predefined constraints—protecting personal data, refraining from unauthorized actions, or staying within boundaries—regardless of immediately observable harm. Harm severity (Sharm) measures the consequences of tasks that do violate constraints. By conditioning only on violating tasks (e.g., an unintended DELETE statement versus a benign sorting error), Sharm separates how bad violations are from how often they occur.

3.5. Aggregation To enable upstream comparisons across agents, we aggregate reliability metrics within each dimension and compute an overall reliability score:

The following aggregation choices merit explanation:

Consistency. We weight outcome, trajectory, and resource categories equally. Aggregate trajectory consistency Ctraj = 1 2(Cd traj + Cs traj) prevents it from dominating overall consistency due to having more sub-metrics. While trajectory consistency is highly valuable in contexts that demand auditability and process reproducibility, we recognize it may be less desirable in open-ended creative workflows. In such cases, practitioners can choose to exclude it from the consistency aggregate.

Safety. The safety score follows the classical risk formulation of Kaplan & Garrick (1981), decomposing risk into the product of violation probability (1 −Scomp) and expected severity (1 −Sharm). Thus, RSaf = 1 only when an agent never violates constraints or causes no harm. We explicitly exclude safety from the overall aggregate because safety violations are inherently a tail phenomenon. Averaging safety with other dimensions obscures critical tail risks (e.g., behaving safely 99% of the time but causing catastrophic harm 1% of the time).

Discussion. Treating reliability as an independent axis of progress has implications for how agents are evaluated, designed, and governed. We summarize four recommendations here and expand on each in Appendix A.

Recommendation 1. Evaluating reliability requires dynamic benchmarks that move beyond single-run accuracy and fixed environments.

Current agent benchmarks typically report a single accuracy number from a single run in a fixed environment. This reveals nothing about whether an agent would succeed on the same task tomorrow, how it handles a rephrased instruction, or how it adapts when its underlying infrastructure shifts. Deployed agents face a different reality: databases are migrated, API formats change, and tool libraries are updated. Reliable measurement therefore requires multi-run protocols that re-execute identical tasks to assess variance, alongside multi-condition protocols that systematically perturb inputs and environments. Benchmarks should become generative and parameterized rather than relying on fixed Towards a Science of AI Agent Reliability test sets, letting experimenters rename fields, reorder responses, or inject fault probabilities that mirror real-world shifts. Such generative test sets also mitigate shortcuts such as looking up answers online (Kapoor et al., 2025). Finally, temporal re-evaluation is essential to reveal whether reliability is maintained or silently degrades as the world drifts from the conditions under which an agent was tested.

Recommendation 2. Agent architectures should be designed and optimized for reliability, not just capability.

Agent design should be explicitly guided by our reliability dimensions, which improve unevenly across model generations: calibration and safety have advanced noticeably, suggesting intentional optimization during training, while consistency and discrimination have improved little. Systematic evaluation makes this uneven progress visible, identifying which dimensions are on a positive trajectory and which are not. Where gains occur, our metrics track them; where they are absent, they provide optimization targets that capability-oriented evaluation alone leaves invisible.

Recommendation 3. Reliability metrics should inform deployment governance, as in safety-critical industries.

Reliability metrics and incident analyses should feed into deployment decisions, change management, and compliance. An organization could require minimum consistency and safety thresholds before promoting an agent from a sandboxed pilot to production, much as aviation systems must meet certification requirements before entering service. Analogous to safety-critical industries, a culture of incident reporting, post-mortem analysis, and continuous improvement will likely prove crucial. Treating reliability as multi-dimensional opens space for research contributions such as metric design, benchmark development, algorithmic methods, interface design, and governance.

Recommendation 4. Reliability requirements should scale with the degree of agent autonomy.

How much reliability matters depends on whether an agent operates autonomously or augments a human. In augmentation settings (coding assistants, search copilots), a human reviews and approves output before it takes effect, serving as a reliability backstop. Notably, this has enabled AI coding assistants to reach wide adoption despite imperfect reliability. In automation settings (customer-service chatbots, unattended workflows), the agent’s output is the final action with no human buffer, so unreliability translates directly into failures. An agent that succeeds on 90% of tasks but fails unpredictably on the rest may be a useful assistant yet an unacceptable autonomous system.

Conclusion. We introduced a decomposition of agent reliability grounded in safety-critical engineering and evaluated 15 models across two complementary benchmarks. Our results show that 24 months of rapid capability gains have produced only small improvements in reliability: models that are substantially more accurate remain inconsistent across runs, brittle to prompt rephrasings, and often fail to understand when they are likely to succeed. As agents are deployed in high-stakes settings, treating reliability as a key evaluation concern becomes essential. We have proposed one way to do so: a four-dimensional decomposition with twelve distinct sub-metrics grounded in safety-critical engineering. While our specific decomposition is one of many possible framings, the core shift in perspective matters most: from asking “How often does the agent succeed?” to asking “How predictably, consistently, robustly, and safely does it behave?”. We expand on our recommendations and limitations in Appendix A and Appendix B, and outline avenues for future work in Appendix E.

Limitations. We acknowledge several limitations of our work:

• Benchmark coverage. Our analysis covers two benchmarks (τ-bench and GAIA) which, while complementary in structure and scope, represent a narrow slice of the tasks agents face in practice.

• Scaffold diversity. We evaluate each benchmark with a single well-performing scaffold; other scaffolds could yield qualitatively different reliability profiles.

• Safety judging. Our safety evaluation relies on LLMbased judging, introducing its own reliability concerns.

• Metric and aggregation choices. The metrics within each dimension, and our normalization-based approach to disentangling reliability from capability, reflect subjective design choices; alternate decompositions are possible. We report safety separately to avoid masking tail risks, so R does not capture the full picture.

• Choice of temperature. We set temperature to zero throughout, which may overestimate the reliability achievable when nonzero temperatures maximize accuracy.

We view our framework as a starting point and stress that reliability evaluation should complement, not replace, careful deployment practices including human oversight, sandboxed testing, and monitoring.