HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks

Healthcare administration accounts for over $1 trillion in annual spending, making it a promising target for LLM-based computer-use agents (CUAs). While clinical applications of LLMs have received significant attention, no benchmark exists for evaluating CUAs on end-to-end administrative workflows. To address this gap, we introduce HEALTHADMINBENCH, a benchmark comprising four realistic GUI environments—an EHR, two payer portals, and a fax system—and 135 expert-defined tasks spanning three administrative task types: Prior Authorization, Appeals and Denials Management, and Durable Medical Equipment (DME) Order Processing. Each task is decomposed into fine-grained, verifiable subtasks, yielding 1,698 evaluation points. We evaluate seven agent configurations under multiple prompting and observation settings and find that, despite strong subtask performance, end-to-end reliability remains low: the best-performing agent (Claude Opus 4.6 CUA) achieves only 36.3% task success, while GPT-5.4 CUA attains the highest subtask success rate (82.8%). These results reveal a substantial gap between current agent capabilities and the demands of real-world administrative workflows. HEALTHADMINBENCH provides a rigorous foundation for evaluating progress toward safe and reliable automation of healthcare administrative workflows. We release the benchmark, environments, and leaderboard at https://healthadminbench.stanford.edu.

Introduction. Large language models (LLMs) in healthcare have primarily been evaluated on clinical tasks such as diagnostic reasoning and triage (Arora et al., 2025; Bedi et al., 2025). In contrast, administrative tasks, including prior authorization, claims submission, and patient intake, remain underexplored despite accounting for over $1 trillion in annual U.S. healthcare spending (Sahni et al., 2023) and contributing to preventable medical errors (Makary & Daniel, 2016; Rodziewicz et al., 2024). These tasks require multi-step interactions across legacy enterprise systems that lack API support, limiting direct programmatic automation.

LLM-based computer-use agents (CUAs), which observe and interact with software interfaces, offer a potential solution. However, existing benchmarks do not capture the multi-step and cross-system complexity of administrative healthcare tasks. Prior work focuses on web navigation tasks on public websites (Zhou et al., 2023; Deng et al., 2023; Garg et al., 2025; Xu et al., 2025) and general enterprise software (Drouin et al., 2024), and does not reflect the constraints of healthcare systems, which involve gated access, limited documentation, and sensitive data. Healthcare-specific efforts partially address this gap. MedAgentBench (Jiang et al., 2025) provides an interactive environment for LLM-based agents, but is restricted to clinical tasks implemented via Fast Healthcare Interoperability Resources (FHIR) API calls and does not support GUI-based interaction. MedHELM (Bedi et al., 2026) includes administrative tasks within a broader evaluation framework, but is limited to static, text-based evaluation and does not capture the dynamic, multi-step environments required for computer-use agents.

To address this gap we introduce HEALTHADMINBENCH, a benchmark for evaluating computeruse agents in healthcare administration. We design multi-step, GUI-based tasks derived from realworld administrative tasks observed through hospital shadowing, and use the “Administration and Workflow” category from MedHELM to guide task selection. We construct four deterministic web environments simulating core administrative systems, including an electronic health record (EHR), two payer portals, and a fax system. Within these environments, we define 135 expert-designed tasks grouped into three administrative task types corresponding to common revenue-cycle workflows: Prior Authorization, Appeals and Denials Management, and Durable Medical Equipment (DME) Order Processing Building on the REAL framework (Garg et al., 2025), we define verifiers (1,177 deterministic and 521 LLM-based) for 1,698 subtasks, enabling fine-grained evaluation of agent behavior. We evaluate seven agent configurations derived from five frontier models (including both our harness and native computer-use agents) and find that the strongest agent achieves a 36.3% full-task success rate. Although agents perform well on individual subtasks, they fail to reliably complete end-to-end tasks due to errors in document handling and cross-system data transfer.

Our contributions are as follows:

1. Interactive environments for healthcare administration. We publish four deterministic computer-use environments to replicate common administrative interfaces, including an Electronic Health Record (EHR), two payer authorization portals, and a fax portal. These environments serve as reproducible proxies for proprietary administrative systems that are not publicly accessible, and can be used for both training and evaluation. 2. Benchmark of 135 specialized administrative tasks. We curate 135 expert-designed, multi-step tasks derived from real-world administrative work and aligned with the Med- HELM ”Administration and Workflow” category. Each task is decomposed into verifiable subtasks using a combination of deterministic checks and LLM-based judges, enabling fine-grained evaluation and reward construction. 3. The first evaluation of computer-use agent reliability in healthcare administration. We evaluate seven agent configurations across five frontier models from multiple providers on HEALTHADMINBENCH and find that the strongest agent achieves only a 36.3% full-task success rate. Although agents often succeed on individual subtasks, they struggle to reliably complete end-to-end tasks, with failures frequently arising from document handling and cross-system coordination. We further show that fine-tuning an open-source model on just 100 training tasks yields a 23% absolute improvement in held-out task success, surpassing the best closed-source agent by 14%. These results suggest that training on these interactive environments may substantially improve agent performance.

To support reproducible benchmarking, we release all environments, task definitions, evaluation code, and model outputs. The public leaderboard is hosted at https://healthadminbench. stanford.edu, while the codebase is available at https://github.com/som-shahlab/ health-admin-bench

Related work. Web interaction. Prior work on web agents spans synthetic environments such as MINIWOB++ (Liu et al., 2018), which focus on basic UI primitives, to more realistic benchmarks such as WEB- SHOP, WEBARENA, and MIND2WEB that introduce longer-horizon, multi-website tasks (Yao et al., 2023; Zhou et al., 2023; Deng et al., 2023). More recent work incorporates multimodal interaction via screenshots (WEBVOYAGER) and deterministic replicas of real-world environments with programmatic evaluation (REAL) (He et al., 2024; Garg et al., 2025).

Enterprise and “workplace” web environments. Recent benchmarks shift toward enterprise workflows with structured schemas and multi-step constraints. WORKARENA and REAL evaluate agents in realistic settings such as CRM and ticketing systems, while frameworks like BrowserGym unify these environments under a common interface for large-scale evaluation (Drouin et al., 2024; Garg et al., 2025; Chezelles et al., 2025). However, these benchmarks focus on general enterprise tasks and do not capture domain-specific workflows such as healthcare administration, which require coordination across multiple systems and organizations.

OS and multi-application workflows. Beyond the browser, benchmarks now test full desktop environments. OSWORLD (Xie et al., 2024) evaluates multimodal agents on open-ended tasks in real OS environments, requiring GUI grounding, tool switching, and recovery from errors. These results suggest that failures often arise from brittle perception-action grounding and incomplete operational knowledge rather than purely “reasoning” mistakes. These challenges are amplified in healthcare administration, where tasks require coordination across multiple environments and reliable execution of schema-constrained interactions.

Table 1 summarizes how these benchmarks compare along dimensions that matter for administrative work, including cross-application workflows, schema-constrained interactions, explicit commit actions, and objective success checks.

Clinical reasoning and dialogue. Benchmarks and evaluation suites such as MultiMedQA (Singhal et al., 2022) and Med-HALT (Pal et al., 2023) evaluate diagnostic reasoning, clinical question answering, and safety vulnerabilities. More recent comprehensive evaluations such as MedHELM (Bedi et al., 2026), assesses LLMs across a broad range of tasks, including clinical decision support, patient communication, documentation, medical research, and healthcare administration. However, these evaluations are largely text-based and do not capture the end-to-end, payer-facing tasks (e.g., portal navigation, form completion, and submission) that underpin revenue-cycle operations.

Method. We define administrative healthcare tasks as sequential decision problems over interactive web environments, formalized as partially observable Markov decision processes (POMDPs). The latent state st ∈S captures the full configuration of all browser sessions (e.g., page content, navigation context, and task-relevant statuses), while the agent observes a partial view ot = O(st), consisting of a screenshot or structured interface representation.

At each timestep, the agent selects an action at ∈A based on the task specification and interaction history, which is executed via a browser automation engine (Playwright) to deterministically transition the environment to st+1 = T(st, at). Episodes terminate upon task completion or when a fixed interaction budget is reached, and performance is measured via a binary terminal reward indicating whether the task was successfully completed.

HEALTHADMINBENCH includes four deterministic web environments that represent core revenuecycle systems: an EHR-like viewer with multiple task-specific views (e.g., prior authorization, denials, and DME worklists), two payer authorization portals with distinct layouts and policies, and a fax submission portal. These environments were designed based on over 100 hours of observation of real administrative tasks, during which staff routinely transfer information across multiple systems.

Each environment is implemented as a sandboxed web application with typed inputs, schema validation, policy checks, and realistic failure modes (e.g., missing required fields). Environments are initialized with synthetic patient data to support task execution. Although both the data and environments are synthetic, they are designed to enforce task logic consistent with real payer policies and documentation requirements. We abstract away deployment-specific access constraints such as CAPTCHAs, MFA, and session timeouts in order to isolate workflow execution in a controlled, reproducible setting. Agents must combine chart interpretation with structured form completion under heterogeneous portal constraints.

The observation function O(st) defines the information available to the agent at timestep t. HEALTHADMINBENCH supports observation modalities that reflect realistic enterprise computer use. Specifically, observations may consist of: (i) a rendered screenshot of the current browser viewport, or (ii) a structured interface representation derived from the accessibility tree, in which interactive elements are indexed by stable data-testid identifiers. This representation provides a controlled, reproducible abstraction for element-level interaction. Although such identifiers are typically not exposed in real-world EHR and payer systems, they approximate the structured grounding available through accessibility layers or instrumentation.

The data-testid-indexed representation follows prior computer-use benchmarks (Garg et al., 2025) and allows agents to reference UI elements symbolically without reasoning over pixel coordinates. This aligns with language-based agents that operate over structured or textual inputs. In contrast, screenshot-based observations require visual grounding, where agents must infer both the correct action and its location on the screen.

We define a finite action space that captures user-level keyboard and mouse operations. Actions are expressed in natural language at the level of user intent (e.g., “click submit”, “enter text”), and are mapped to executable browser operations. For agents with access to the accessibility tree, actions operate on interface elements addressed by their data-testid identifiers, enabling symbolic selection and manipulation of UI components. Agents with screenshot-only observations use the same action set but must additionally infer the spatial location of each action.

The action space includes standard browser interactions such as clicking buttons, entering text, selecting dropdown options, toggling checkboxes, uploading or downloading files, and scrolling. A complete enumeration of supported actions is provided in Table 5.

Discussion. HEALTHADMINBENCH addresses a key gap in the evaluation of healthcare AI agents by moving beyond static, text-only assessments to realistic administrative workflows that require long-horizon, cross-system interaction. Across all evaluated agents, end-to-end task success remains low—the strongest agent (Claude Opus 4.6 CUA) reaches only 36.3%—despite much higher subtask success rates. This gap suggests that current computer-use agents can often execute individual actions correctly, yet still struggle to reliably complete full administrative workflows, particularly when tasks require document handling, cross-portal coordination, and sustained state tracking over many steps. At the same time, our fine-tuning results suggest that these failures are not solely a consequence of model capability, but also reflect the scarcity of domain-specific workflow data and training environments.

Conclusion. We introduce HEALTHADMINBENCH, the first benchmark for evaluating computer-use agents on realistic healthcare administrative workflows. Our results show that, despite promising subtask-level performance, current frontier agents remain far from reliable on end-to-end tasks, revealing a substantial gap between interface-level competence and deployment-ready workflow automation. By releasing reproducible environments, expert-grounded tasks, and fine-grained evaluation protocols, we aim to provide a foundation for measuring progress toward safer and more effective administrative AI systems in healthcare.

Limitations. While our environments and task set capture key sources of structural difficulty in healthcare administrative workflows, they remain simplified proxies for production systems. In particular, the benchmark is static and retrospective: it does not capture longitudinal interface drift, evolving payer policies, or the full heterogeneity of real hospital deployments. Although the accessibility-tree setting enables controlled and reproducible evaluation, real-world portals do not typically expose such clean, stable identifiers, and frequent UI updates may reduce the robustness of agents that rely on them. Agents also operate under limited context windows, we evaluate single-shot trajectories without human-like recovery behavior, and we abstract away deployment constraints such as MFA, CAPTCHAs, and session timeouts. More broadly, while administrative automation offers the potential to reduce cost and burden, premature deployment of agentic systems risks operational errors with financial or clinical consequences, underscoring the need for rigorous evaluation and human oversight.