How do interaction signals reshape what generative AI actually receives?

Prior taxonomies of how text prompts combine with GUI interactions (clicks, drags, brushes, sketches) in generative-AI interfaces are, in the authors' framing, "mostly descriptive" and hard to use for comparing or generating new designs — Gao et al.'s and Hu et al.'s paradigm lists (GenAI as medium, tool, partner, mediator) describe what exists but not how the pieces combine, and Riche et al.'s AI-Instruments principles (Reification, Reflection, Grounding) "enrich the considerations for generating new interaction designs but fall short in comparing multiple designs." Shen, Wang, Qu, Xie, and Li's Interaction-Augmented Instruction (IAI) model addresses the gap with a compact entity–relation graph formalizing how the combination of interactions and text prompts enhances human-GenAI communication. It defines six entities — Human (H), Interaction (I), Artifact (A), Text Prompt (T), Augmented Instruction (Aug), and Generative AI (G) — and from applying the model to existing tools, the authors "distill twelve recurring and composable atomic interaction paradigms."

The model is derived from two base interaction paths: the canonical prompt-driven flow H→T→G→A, and the general GUI-interaction flow H→I→A. Three distinctions build the model from there. First, T (free-form natural language) is separated from I (focused, non-textual actions like a brush mask, bounding box, or selected code block) because the two "differ sharply in expressivity, precision, and in how the model understands them." Second, Aug is introduced as a new entity because prompts and interactions "are not independent 'inputs' that the model somehow interprets in isolation" — interaction produces structured constraints (pixel masks, coordinate ranges, AST node identifiers) that must be "encoded, normalized, and attached to a prompt in a machine-readable form" before reaching G. Third, auxiliary entities are folded in to keep the model parsimonious: context inputs (retrieved passages, uploaded files) become part of Artifact, and transient interaction products (highlighted spans, intermediate masks) become manifestations of Interaction rather than separate entities. The paper's argument is that without Aug as an explicit entity, designs that combine T, I, and A differently "collapse into the broad 'non-text interactions' category" — Aug is what lets the model tell two superficially similar interfaces apart.

This sits alongside How should generative UI be organized as a design space? as a second HCI-community attempt to formalize the prompt/interaction/GenAI design space into comparable units, but at a different grain: that workshop proposal sorts genUI by audience, constancy, control, and timeframe at the level of the deployed interface, while IAI works one level down, graphing what actually gets composed into a single model call. It also echoes How should AI interfaces handle the shift from doing to supervising? in treating structured, non-linguistic specification as a necessary complement to language, though Nielsen's layers describe system architecture for negotiating intent while IAI's entities describe how an instruction gets assembled per turn. The claim that interaction signals "collapse referential ambiguity and materially improve the likelihood of a targeted, single-turn outcome" bears on How do users actually form intent when prompting AI systems?: IAI treats Aug as resolving referential ambiguity within a turn, while that note's framing treats intent itself as evolving across turns — together they point to referential grounding and intent evolution as separate problems an interface may need to solve at once, not one problem with two names.

The twelve paradigms and the model itself come from "a UI-centric corpus of 66 tools" examined through "qualitative synthesis and illustrative case studies rather than empirical user or designer studies," and the authors state the scope covers only "single-user, single-agent interactions," not coordination among multiple GenAI agents or shared augmented instructions. The evaluation axes the model proposes — referential fidelity, convergence cycles, controllability, provenance clarity — are offered as implications of the model, not measured against the prior taxonomies it argues are weaker. What the excerpt establishes is a more expressive vocabulary for naming differences among existing interfaces, not evidence that designing with that vocabulary produces interfaces people actually find easier to use.