Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration

Text prompt is the most common way for human-generative AI (GenAI) communication. Though convenient, it is challenging to convey fine-grained and referential intent. One promising solution is to combine text prompts with precise GUI interactions, like brushing and clicking. However, there lacks a formal model to capture synergistic designs between prompts and interactions, hindering their comparison and innovation. To fill this gap, via an iterative and deductive process, we develop the Interaction-Augmented Instruction (IAI) model, a compact entity–relation graph formalizing arXiv:2510.26069v3 [cs.HC] 24 Feb 2026 how the combination of interactions and text prompts enhances human-GenAI communication. With the model, we distill twelve recurring and composable atomic interaction paradigms from prior tools, verifying our model’s capability to facilitate systematic design characterization and comparison. Four usage scenarios further demonstrate the model’s utility in applying, refining, and innovating these paradigms. These results illustrate the IAI model’s descriptive, discriminative, and generative power for shaping future GenAI systems.

Introduction. Generative AI (GenAI) has rapidly become a general-purpose technology, enabling new intelligent applications across domains such as design [70], education [39], and data analysis [32]. Modern large language and multimodal models can interpret open-ended text prompts and generate diverse outputs (e.g., text, image, video), making the text prompt interface the dominant paradigm for human–AI communication [72]. Yet free-form text is often too ambiguous and coarse to express fine-grained or nuanced intent [64, 71, 78]. For instance, in image editing, a user might want to modify “the upper-right flower petal”, but a text prompt alone is hard to accurately localize that target visual element. Likewise, in coding, asking “write a loop to process this data” could imply different iteration strategies. Correspondingly, in practice, users often struggle to articulate sufficient detail or to break down complex goals into a single prompt. To address these limitations, a promising strategy is to combine general-purpose but imprecise text prompts with dedicated interactions (e.g., clicks, drags, brushes, etc.) through graphical user interfaces (GUIs), as shown in Fig.1-A. The synergy enables both flexible and fine-grained user control. For example, consider Jason, a creative director designing a stylized magazine cover portrait. His intent is clear: “I need a stylized portrait for a magazine cover, with high fashion and artistic lighting”. Compared with the basic prompt-only workflow (Fig.2-A), this task can be realized through a spectrum of interface designs (Fig.2 B–F) that combine prompts and interactions in distinct ways. Concretely, some interfaces let him refine or expand the text prompt before generation (Fig.2-B), for instance, selecting “artistic lighting” for fine-tuning or switching to a “minimal black-and-white” style via UI controls. This functionality speeds up exploration but still assumes all intent can be expressed in language [1, 12]. To go beyond language, Jason might manually sketch a portrait outline or brush key regions, merging these non-linguistic signals with the prompt for more precise spatial and referential control (Fig.2-C) [37, 45]. When an initial render already exists, he may directly draw a necklace on the portrait, with the system incorporating both the prompt and the manipulated artifact content to enable precise, auditable edits grounded in the actual image (Fig.2-D) [49, 55]. After generating an initial image, some systems propose interactive controls (Fig.2-E), such as sliders for keyword (e.g., “stylized” and “high fashion”) attention or color palettes for finer “artistic lighting”, allowing Jason to steer subsequent generations without rewriting text [28, 88]. Others analyze the generated artifact and propose structured options (Fig.2-F): Jason can choose to generate cover text, select the type, style, and length, and assemble a structured instruction for a magazine tagline [36, 98]. Although these designs support the same creative goal, they differ fundamentally in various dimensions, such as where user intent is specified (e.g., text, interaction, artifact), when interactions occur (pre- or post-model invocation), and what information is ultimately delivered to GenAI. Those structural differences affect referential fidelity, controllability, provenance, and user experience. Furthermore, such strategies have not yet been formalized as a comprehensive interaction model to guide related research and practices. As a pioneering study in the HCI field notes [9], these “point-like” research demonstrations of the strategy are often insufficient due to the challenges of transforming task-specific demonstrations to broader real-world applications and understanding the advancement of related techniques. It is essential to propose general interaction models to facilitate the careful examination of existing designs (descriptive power), the comparison between design variations (discriminative power), and the consequent innovation of new interaction designs for emerging tasks (generative power) [8, 9, 44]. Towards this goal, there has been a series of previous research that attempted to reveal and advance the interaction-text prompt synergy. Gao et al. [25] and Hu et al. [31] have developed taxonomies of interaction paradigms for interactive GenAI applications, such as GenAI as a medium, tool, partner, and mediator. However, the taxonomy is mostly descriptive and is difficult to use to guide the generation of new designs for new scenarios. Riche et al. [64] proposed AI-Instruments, extending instrumental interaction [8] by treating prompts themselves as manipulable interface objects, guided by three principles: Reification, Reflection, and Grounding. These principles enrich the considerations for generating new interaction designs but fall short in comparing multiple designs within a model that explicitly represents the interplay between text prompts and other interaction modalities.

Related work. Prior work on human–GenAI collaboration covers communication paradigms, design patterns, and theoretical models; we review these to highlight progress and motivate a unified model of interactionaugmented instructions.

Human–GenAI communication spans a spectrum of interface strategies, from free-form text prompting to fully encapsulated tools and hybrid paradigms that combine prompting with GUI interactions. Free-form text prompts remain the dominant way for human–GenAI communication due to their flexible, expressive, and low-friction [21, 91]. However, prompt-only interaction has well-documented limitations: natural language is often ambiguous, underspecified, or ill-structured for task-specific operations, which increases iteration cost and reduces control [64, 71, 78]. Empirical studies highlight recurring challenges such as prompt formulation, disambiguation, and intent steering—for example, users struggle to localize visual elements precisely through language alone [23, 55] or to specify the exact semantic behavior desired in code edits [22, 95]. Another line of interface design has been to encapsulate GenAI capabilities within backend agents or domain-specific tools [7]. In such designs, the system exposes a constrained interface (e.g., widgets, templates, or menus) [46, 54, 85, 87] or totally automate the process while GenAI executes domain logic behind the scenes [67, 69, 73]. For example, Textoshop [54] allows text editing entirely through drawing software-like interactions, with prompts handled internally; fully automated agents similarly hide prompting from the user [69, 99]. This lowers user burden but constrains openness and the expressive flexibility of prompting. A more flexible and increasingly prevalent approach is to combine prompting with GUI interactions—what we call interactionaugmented instructions. Here, prompts are enriched by targeted user actions such as clicking, brushing, sketching, or selecting [25, 31, 51, 71, 86]. These interactions may occur before the GenAI invocation, e.g., selecting elements to constrain the scope of a prompt [19, 23, 48, 49, 55, 61, 84, 93, 96], organizing multiple prompts into structured forms such as trees or graphs [22, 28, 91, 95], sketching to guide content generation [37, 45, 59, 100], annotating text to extend context or highlight specific information [68], or after it, e.g., clarifying user intent through follow-up queries [3, 6, 13, 28, 52], presenting suggestions for human confirmation [4, 26, 38, 90], converting prompt into structured GUI components [14, 97], or enabling direct post-generation artifact manipulation [27, 33, 62, 79, 82, 92]. Together, these paradigms illustrate diverse strategies for balancing the openness of prompting with the precision of interaction.

Method. To systematically capture the synergy between text prompts and interactions in GenAI system interfaces, we propose the Interaction- Augmented Instruction Model, which is a formal, entity–relation graph that unifies general GUI interaction concepts with new constructs introduced by GenAI (Fig.3). It distills the essential components of human–GenAI communication and their relations, enabling descriptive, discriminative, and generative analysis of diverse interfaces. This section first describes how we derive the key model concepts, i.e., entities (Sec. 3.1) and relations (Sec. 3.2), and then discusses applying these concepts to represent an interface design with directed graphs (Sec.3.3).

3.1 Entities The IAI Model comprises six core entities: Human (H), Interaction (I), Artifact (A), Text Prompt (T), Augmented Instruction (Aug), and Generative AI (G). Concretely, we treat an entity as a semantically coherent object or agent in the interaction ecology. Tab.1 provides an overview for all entities and relations. We derived the entities through an iterative, deductive process that begins with asking a simple question: what are the irreducible elements that appear in the common prompt-driven and interactiondriven paths in practice? Two empirically ubiquitous paths served as the starting point. The first is the canonical prompt-driven flow, H →T →G →A, in which a human (H) composes a text prompt (T) that the GenAI (G) model executes to produce or modify an artifact (A) [41, 64]. The second is the general GUI interaction flow, H →I →A, in which a human (H) leverages interactions (I) on GUIs to act on an artifact (A) [8, 74]. Based on the two interaction paths, three distinctions are critical to derive our model. First, to capture the qualitative differences between modalities, we separate Text Prompt (T) from Interaction (I). Conceptually, T is a free-form, general-purpose, natural language specification of intent (e.g., “a stylized portrait of a young woman” in Fig.2-A). I denotes focused, non-textual, often GUI user actions (e.g., click in Fig.2-B, sketch in Fig.2-C, brush in Fig.2-D, drag in Fig.2-E, widget selection in Fig.2-F) that can modify, supplement, or operationalize concrete referents or constraints (e.g., a brush mask, a bounding box, a selected code block). This separation matters because T and I differ sharply in expressivity, precision, and in how the model understands them. However, such separation is not sufficient to represent what the model actually consumes in many practical GenAI workflows. To address the issue, secondly, we introduce Augmented Instruction (Aug) as a new entity. In GenAI system interfaces, text prompts and interactions are not independent “inputs” that the model somehow interprets in isolation. Interactions often produce structured, non-linguistic constraints (e.g., pixel masks, coordinate ranges, AST node identifiers, selected table rows or text segments) that must be encoded, normalized, and attached to a prompt in a machine-readable form [1, 21, 47, 49, 68]. Viewing Aug as the explicit input to GenAI simplifies paradigm comparisons, as tools differ mainly in how they combine T, I, and A to build Aug.

Third, during iteration we consolidated several auxiliary entities (found in Sec.2.2) into the six core entities to keep the model parsimonious while maintaining expressiveness. Concretely: (1) Context inputs (e.g., retrieved passages, prior conversation state, uploaded reference files) are modeled as part of Artifact (A) because they function as domain objects that ground instructions [25]. Their inclusion is modeled via A →Aug (artifact-derived constraints) or A →G (artifact supplied as raw model context). (2) Temporary interaction products (e.g., highlighted spans, sketched regions, intermediate masks, or annotation buffers) are treated as manifestations of Interaction (I) rather than independent entities [31]. They have only transient semantic life: they exist to constrain or point to artifacts or prompts and then are encoded into instructions or discarded.

Discussion. Why do we need instruction-augmented interaction? We model the interplay between prompts and interactions because GenAI systems increasingly rely on both, yet lack a framework to make their complementarity explicit. The IAI model and the twelve atomic paradigms articulate a core claim: prompts and focused interactions are complementary (not interchangeable) modes for externalizing user intent. Treating Augmented Instruction (Aug) as the explicit input to generative models foregrounds a functional separation: Text Prompt (T) supplies high-level, abstract goals; Interaction (I) supplies precise, referential constraints and grounding. There are three closely related reasons this interplay is necessary. First, generative models map underspecified instructions to a broad set of plausible outputs [64]; interaction signals collapse referential ambiguity and materially improve the likelihood of a targeted, single-turn outcome [65]. Second, interactions encode provenance and manipulable constraints that support fine-grained control properties that language alone cannot reliably provide at scale [75]. Finally, the IAI model brings human–GenAI communication closer to human–human interaction. In practice, people rarely rely on language alone; they complement speech with gestures, sketches, and shared artifacts to ground intent. By mirroring these multimodal practices, IAI reduces ambiguity and enriches expression, pointing toward GenAI systems that act as more natural collaborators in human–AI co-creation.

How should we apply interaction-augmented instructions? Adopting interaction-augmented instructions implies several foundational shifts in human–AI practices. First, expertise will shift from prompt engineering to instruction design. Users should compose reusable and multimodal instructions rather than optimizing text alone, while tools need to expose these capabilities with clear discoverability to support effective use [77]. Second, cognitive load and UX trade-offs must be managed: interactions can reduce text specification but add interface complexity, so designers should prioritize low-friction affordances (e.g., defaults, progressive disclosure, previews) to minimize overhead [78]. Third, tool evaluation can broaden beyond artifact quality to include axes that follow directly from the IAI distinction, such as correctness of local changes tied to interactions (referential fidelity), rounds to satisfactory output (convergence cycles), ease of steering behavior (controllability), and user comprehension of influences (provenance clarity).

What can our model contribute to interaction-augmented instructions? The IAI model contributes to both engineering practice and HCI theory-building. It provides greater descriptive, discriminative, and generative power than methods in prior work, such as Gao et al. [25] and Shen et al. [71], by introducing two innovations. First, the model explicitly encodes information flow in the paradigm graph. Each edge denotes a precise link between entities, making differences in timing, source, and composition of instructions formally visible. Another key contribution is treating Aug as an explicit instruction to GenAI for the expressiveness of the model. Without it, designs that combine text prompts, interactions, and artifacts in different ways collapse into the broad “non-text interactions” category. It cannot systematically distinguish important differences, CHI ’26, April 13–17, 2026, Barcelona, Spain Leixian Shen, Yifang Wang, Huamin Qu, Xing Xie, and Haotian Li such as that between P1 and P2. With Aug, these differences become structurally visible, varying how entities feed into GenAI, and in what order, systematically yielding distinct paradigms.

Conclusion. We introduced the Interaction-Augmented Instruction (IAI) model that formalize how natural language prompts and GUI interactions jointly shape human–GenAI communication. With the IAI model, we summarized twelve atomic paradigms based on existing tools, which provide reusable abstractions that enable systematic characterization, comparison, and innovation in interface design. Our usage scenarios demonstrate how this model and the extracted twelve atomic paradigms bridge conceptual models with actionable design choices, supporting refinement of existing tools and exploration of new interaction spaces. The paradigms and usage scenarios jointly verify that our proposed IAI model have sufficient descriptive, discriminative, and generative power to model the interplay of prompts and GUI interactions. In future, we plan to further explore this research direction through extending the model to finer granularity, expanding the collection of paradigms, and exploring more usage scenarios for the model and paradigms, such as generative UI.

Limitations. Our contributions should be interpreted within several boundaries. First, the IAI model and the analyzed atomic paradigms currently focus on single-user, single-agent interactions and do not yet formalize coordination among multiple GenAI nodes or the construction of shared augmented instructions. Second, the taxonomy is derived from a UI-centric corpus of 66 tools, which, although diverse, cannot fully represent the rapidly evolving GenAI ecosystem. Third, validation relies on qualitative synthesis and illustrative case studies rather than empirical user or designer studies.