In real hospitals, does an AI agent that talks directly to software beat one that just clicks around like a human?
Do API-based agent interactions outperform GUI-based approaches in real healthcare systems?
This explores whether AI agents that work through software interfaces directly (APIs) do better than agents that click through screens like a human would (GUI-based), specifically in hospital and health-system settings.
This explores whether agents that call software directly through APIs beat agents that click through screens, in real healthcare systems. The short answer: the corpus doesn't contain a head-to-head API-vs-GUI test in healthcare. What it does have are two separate pieces of evidence that point the same way when you put them side by side.
The first piece comes from outside healthcare. The AXIS framework found that agents which call APIs first, and only fall back to step-by-step clicking when they have to, finish tasks 65–70% faster with 97–98% accuracy and much less cognitive load on the people supervising them Can API-first agents outperform UI-based agent interaction?. AXIS also deals with the obvious objection that most legacy software has no clean API: the agent explores existing applications and builds APIs for itself. That matters a lot for hospitals, whose software stacks are full of old systems that were never designed to be automated.
The second piece shows how screen-clicking agents do in healthcare today. HealthAdminBench gave frontier computer-use agents 135 real administrative workflows, such as prior authorizations and claims work. The best agent finished only 36.3% of tasks end to end, even though it got 82.8% of individual steps right Can AI agents reliably complete healthcare administrative workflows?. That gap is the real story. Small per-step errors add up over long workflows, and the failures cluster in document handling and coordination across systems, which are the parts where an agent has to read screens and move between applications. Part of the reason is that screen navigation is genuinely hard perception work: general multimodal models lack the grounding to act reliably on real interfaces, and agents that read HTML or accessibility trees miss what's actually on the screen Do text-based GUI agents actually work in the real world?. Combining the two studies, a reasonable hypothesis is that APIs would remove exactly the failure points HealthAdminBench found. That is an inference, though, not a tested result.
The less obvious lesson is that the interface may matter less than what you build around the model. Fine-tuning an open-source model on just 100 healthcare admin tasks raised its success rate by 23 points and put it ahead of the best closed model Can domain-specific fine-tuning close healthcare agent reliability gaps?. In clinical diagnosis, wrapping o3 in an orchestration layer slightly improved accuracy while cutting the cost per case by about 70%, and the gains carried over to other model families Can orchestration strategies boost diagnostic AI without better models?. Both results fit a broader argument: reliable agents come from moving memory, skills, and structured protocols out of the model and into the surrounding system Where does agent reliability actually come from?. An API is one such protocol. Seen this way, "API vs GUI" is a special case of a bigger question: how much of the workflow's structure do you hand the agent, and how much do you make it work out from pixels?
If you want to go deeper, read the HealthAdminBench results next to AXIS. One shows where screen-clicking agents break in healthcare, and the other shows what changes when you stop making agents click. The experiment that would settle the question, API agents run on HealthAdminBench-style workflows, isn't in the collection yet.
Sources 6 notes
The AXIS framework shows that prioritizing API calls over sequential UI interactions cuts task completion time by 65–70% while maintaining 97–98% accuracy and reducing cognitive workload by 38–53%. A self-exploration mechanism automatically discovers and constructs APIs from existing applications, solving the bootstrapping problem.
HealthAdminBench tested frontier AI agents on 135 healthcare administrative workflows and found the best achieved only 36.3% full-task success despite 82.8% subtask accuracy, with failures concentrated in document handling and cross-system coordination.
ShowUI demonstrates that GUI agents need end-to-end vision-language-action models with UI-aware token selection and interleaved streaming, not adapted general-purpose MLLMs. Standard multimodal models lack the grounding and action capabilities real interface navigation demands.
Fine-tuning an open-source model on 100 healthcare administrative tasks increased held-out success by 23 percentage points, surpassing the best closed-source agent by 14 points. This suggests agent reliability gaps stem partly from lacking domain training data, not model capability alone.
On 304 NEJM cases, MAI-DxO orchestration achieved 79.9% accuracy at $2,397 per case versus 78.6% at $7,850 for o3 alone. The gains transferred across model families, suggesting the benefit comes from the scaffold, not model weights.
Show all 6 sources
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks
- Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
- Mapping the Design Space of User Experience for Computer Use Agents
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- Towards a Science of AI Agent Reliability
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Sequential Diagnosis with Language Models
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent