AI can skip the interface entirely and just do the task — so when does building a UI still make sense?
How does API-first interaction compare to generative interface approaches?
This explores two ways of getting work done with an AI system — having it call APIs directly under the hood versus having it build a custom interface for you to interact with — and what the corpus says about when each wins.
This explores two ways of getting work done with an AI system: API-first interaction, where the agent skips the visible interface and calls an application's functions directly, versus generative interfaces, where the AI builds a task-specific UI on the fly for a person to use. They sound like rivals, but the corpus suggests they're answering different questions — one optimizes for an agent acting *for* you, the other for you acting *through* the AI.
The case for API-first is efficiency. The AXIS framework shows that when an agent prioritizes API calls over clicking through screens step by step, task completion time drops 65–70% while accuracy holds at 97–98%, and it even includes a self-exploration trick that builds APIs out of existing apps so there's nothing to bootstrap Can API-first agents outperform UI-based agent interaction?. The whole GUI-agent literature is, in a sense, an argument for why you'd want to avoid the screen if you can: vision-only agents stumble when forced to read an interface and decide an action at once Why do vision-only GUI agents struggle with screen interpretation?, text-based agents using HTML or accessibility trees miss what humans actually see Do text-based GUI agents actually work in the real world?, and even the better designs lean on structured, language-centric interfaces rather than raw pixels Can structured interfaces help language models control GUIs better?. API-first sidesteps all that fragility by talking to the machine in the machine's own terms.
Generative interfaces pull the opposite direction — toward the human. Here the finding is that users prefer LLM-generated dashboards, tools, and interactive widgets over walls of text in more than 70% of cases, especially for dense or structured work, because a visible structure lowers cognitive load and invites refinement Do generated interfaces outperform text-based chat for most tasks?. This matters precisely because people often *can't say what they want up front*: intent matures through interaction, and an interface that offers concrete options turns open-ended envisioning into the much easier task of picking from choices Why can't users articulate what they want from AI?. A generated UI is a place for that back-and-forth to happen.
So the real contrast isn't speed versus prettiness — it's where the human sits. API-first removes the human from the loop to go fast; generative interfaces put a richer loop *around* the human to help them figure out what they're even asking for. Both are responses to the same underlying shift: generative AI moves us from specifying *methods* to specifying *intent*, which produces unpredictable outputs and breaks the old consistency assumptions of UI design How should users control systems with unpredictable outputs?. The piece that ties them together is something quieter — command generation. Reframing understanding as generating commands in a domain-specific language, rather than classifying intents, gives you something an API can execute *and* something a generated interface can expose Can command generation replace intent classification in dialogue systems?.
The thing you might not have expected to want to know: these approaches converge. The deepest constraint behind both is that AI's working context is mutable and ephemeral in a way traditional software context never was — users can't internalize it the way they learn a fixed UI, which is what makes naked API calls opaque and on-the-fly interfaces necessary How does AI context differ from conventional software context?. The likely future isn't choosing one. It's agents using APIs to *do* the work and generating interfaces to *negotiate* what the work should be.
Sources 9 notes
The AXIS framework shows that prioritizing API calls over sequential UI interactions cuts task completion time by 65–70% while maintaining 97–98% accuracy and reducing cognitive workload by 38–53%. A self-exploration mechanism automatically discovers and constructs APIs from existing applications, solving the bootstrapping problem.
Research shows users strongly prefer LLM-generated interactive interfaces—dashboards, tools, animations—over text blocks, especially for structured and information-dense tasks. Structured representation and iterative refinement reduce cognitive load.
Agent S's dual-input design—visual input for environmental understanding plus image-augmented accessibility trees for grounding—achieved 9.37% improvement over baseline by factoring planning and grounding into separate optimization paths rather than forcing end-to-end prediction.
OmniParser demonstrates that GPT-4V fails when forced to simultaneously identify icon meanings and predict actions from raw screenshots. Pre-parsing screenshots into structured semantic elements with descriptions lets the model focus solely on action prediction, removing the composite-task bottleneck.
ShowUI demonstrates that GUI agents need end-to-end vision-language-action models with UI-aware token selection and interleaved streaming, not adapted general-purpose MLLMs. Standard multimodal models lack the grounding and action capabilities real interface navigation demands.
Show all 9 sources
Intent develops through interaction, not in isolation. Since AI models respond rather than probe, they miss opportunities to help users discover unarticulated requirements. Structured dialogue that presents model-generated options shifts the cognitive burden from open-ended envisioning to constrained evaluation.
Generative AI shifts interaction to intent specification rather than method specification, creating unpredictable outputs that violate traditional consistency heuristics. Six design principles—including co-creation, imperfection tolerance, and mental model support—address this novel paradigm.
Rasa's dialogue understanding architecture generates domain-specific commands instead of classifying intents, eliminating annotation requirements, handling context naturally, and scaling without degradation—treating understanding as pragmatics rather than semantics.
AI interactions operate on a substrate of constantly shifting context—prompt, history, retrieved data, hidden state—that users cannot internalize like traditional UIs. This structural mutability demands a new design discipline centered on context engineering rather than interface design.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Bridging the gulf of envisioning: Cognitive design challenges in llm interfaces.
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent
- OmniParser for Pure Vision Based GUI Agent
- Turn Every Application into an Agent: Towards Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents
- Large Language Model-Brained GUI Agents: A Survey
- BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent
- Agent S: An Open Agentic Framework that Uses Computers Like a Human
- Large Language Models for User Interest Journeys