Should apps be organized around what you're trying to do, not a fixed menu of buttons and screens?
Should design systems prioritize verbs and actions over static UI elements?
This explores whether interfaces, especially in the AI era, should be built around what people (and agents) are trying to do rather than around a fixed set of buttons, panels and components. The corpus doesn't use the verbs-versus-nouns framing directly, but several lines of research point the same way.
This explores whether interface design should be organized around actions and intentions rather than a fixed library of static UI parts. The corpus never frames it as 'verbs vs. nouns.' Still, a clear pattern runs through it: in AI systems, the weak point is almost always the action layer, meaning what the user wants to do and what the system can actually carry out, not how the screen looks.
Start with the human side. Chat interfaces make users type out their intent, which is slow and lossy. Friedman proposes design levers built around actions: capturing intent, shaping output, refining, delegating tasks, and embedding AI into workflows (Can task-oriented UI patterns reduce the friction of AI chat interfaces?). The deeper problem is that people often don't know what they want until they've interacted with something (Why can't users articulate what they want from AI?). That points to interfaces that offer possible actions to react to, not blank fields to fill in. Generative UI pushes this furthest. When an LLM builds a task-specific interface on the fly, users prefer it to plain chat in over 70% of cases (Do generated interfaces outperform text-based chat for most tasks?). Moran argues this works because the AI decides which interface fits the task, so users who lack 'interface literacy' never have to specify components (Can generative UI serve users who cannot specify interfaces?). In that model, the stable thing is the task, and the screen is disposable.
The twist is that generative tools are weakest at exactly this action layer. A benchmark of generative UI tools found they skipped about a quarter of stated design rationales overall. For functional requirements, the parts that describe what the interface must *do*, the miss rate rose to 34% (Do generative UI tools actually implement their stated design rationales?). The tools produce convincing-looking screens and quietly drop the behavior. That is a strong argument for design systems to write down actions and outcomes explicitly. If you only specify components, an AI generator will happily give you the components and nothing else.
AI agents that operate interfaces add a second reason. Vision-only agents struggle when they have to work out both what an icon means and what to do with it from a screenshot. Pre-parsing the screen into labeled, actionable elements removes that bottleneck (Why do vision-only GUI agents struggle with screen interpretation?), and structured accessibility trees likewise improve planning (Can structured interfaces help language models control GUIs better?). Interface heuristics aimed at what an agent can perceive and control raised task completion without hurting usability for people (Can interface design help agents complete computer tasks better?). In practice, a design system that clearly exposes 'what can be done here' serves two audiences at once.
The counterweight is that 'actions over static elements' can't simply mean abandoning stable structure. AI context keeps shifting, and users can't internalize it the way they learn a fixed menu (How does AI context differ from conventional software context?). Lam and colleagues argue the design decisions between AI and interface should stay visible rather than vanish into generation (Does visible design structure keep HCI expertise necessary as AI advances?). The corpus's answer is closer to 'make actions the primary unit, and keep the structure that tells people where they are and what just happened legible.' It doesn't support either-or.
Sources 10 notes
Friedman identifies five design levers—intent capture, output shaping, refinement, agentic tasks, and workflow embedding—that sidestep slow typed prompts. Examples include node-based intent visualization, style lenses for output, and canvas manipulation, shifting AI interfaces away from conversational chat toward direct interaction.
Intent develops through interaction, not in isolation. Since AI models respond rather than probe, they miss opportunities to help users discover unarticulated requirements. Structured dialogue that presents model-generated options shifts the cognitive burden from open-ended envisioning to constrained evaluation.
Research shows users strongly prefer LLM-generated interactive interfaces—dashboards, tools, animations—over text blocks, especially for structured and information-dense tasks. Structured representation and iterative refinement reduce cognitive load.
Moran argues genUI's advantage is that AI acts as designer, deciding what interface to build rather than requiring users to specify one—addressing the gap where most people lack the interface literacy that vibe coding presupposes.
A benchmark of 24 tasks across five tools found roughly 25% of design rationales go unimplemented, rising to 34% for functional requirements. Tools recognized only half the UX principles embedded in prompts.
Show all 10 sources
OmniParser demonstrates that GPT-4V fails when forced to simultaneously identify icon meanings and predict actions from raw screenshots. Pre-parsing screenshots into structured semantic elements with descriptions lets the model focus solely on action prediction, removing the composite-task bottleneck.
Agent S's dual-input design—visual input for environmental understanding plus image-augmented accessibility trees for grounding—achieved 9.37% improvement over baseline by factoring planning and grounding into separate optimization paths rather than forcing end-to-end prediction.
UI-Verse benchmark testing found that augmented heuristics targeting agent perception, grounding, and control consistently improved agent task completion and efficiency. Human studies confirmed these designs preserved usability for human users with no observable regressions.
AI interactions operate on a substrate of constantly shifting context—prompt, history, retrieved data, hidden state—that users cannot internalize like traditional UIs. This structural mutability demands a new design discipline centered on context engineering rather than interface design.
Lam and colleagues argue that just-in-time objectives architecture makes design decisions between AI and interfaces visible, ensuring HCI expertise remains critical. By surfacing generation-and-evaluation steps at every interaction rather than hiding them, the architecture structures ongoing design judgment into the system itself.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Tools
- Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration
- Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents
- What does Generative UI mean for HCI Practice?
- Generative UI: LLMs are Effective UI Generators
- Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz
- The Articulation Barrier: Prompt-Driven AI UX Hurts Usability
- ShowUI: One Vision-Language-Action Model for GUI Visual Agent