Why can't an AI just treat every click, slider, or drag as if you'd typed it in words?

This explores why a model of human–AI interaction can't treat everything a user does as just more prompt text. The clearest answer in the collection comes from the IAI model, which separates what the user types from what the AI actually receives How do interaction signals reshape what generative AI actually receives?. When you drag a slider, pick a style preset or highlight part of an image, the interface turns that action into constraints and combines them with your prompt into an "Augmented Instruction." A prompt says what you want. A GUI action often fixes a scope, a format, or a limit on what can change. If you fold both into one blob of text, you can no longer compare tools by how they split work between words and widgets. Keeping them apart is what let the authors find twelve recurring interaction patterns across 66 tools, which earlier taxonomies couldn't do.

The reason this matters is that the two channels are good at different things. Friedman argues that typing out your intent is slow and lossy, and that interface moves like node graphs, style lenses and direct canvas edits get intent across faster than prose can Can task-oriented UI patterns reduce the friction of AI chat interfaces?. Research on generated interfaces points the same way: when an LLM builds a task-specific UI instead of replying with text, users prefer it in over 70% of cases, mostly for structured, information-dense work Do generated interfaces outperform text-based chat for most tasks?. If a GUI action were just a prompt in disguise, these gains wouldn't show up. They show up because structure carries information that text has to work hard to express.

The same lesson shows up from the other side, in AI agents that operate GUIs themselves. OmniParser found that GPT-4V struggles when it has to work out what each icon means and decide what to click at the same time. Parsing the screen into labeled elements first lets the model focus only on choosing the action Why do vision-only GUI agents struggle with screen interpretation?. Agent S does something similar: it splits planning (what to do, reasoned in language) from grounding (where on screen to do it, using accessibility trees) and optimizes each separately Can structured interfaces help language models control GUIs better?. UI-Verse goes further and shows that redesigning the interface itself for agent perception improves agent performance without making it worse for humans Can interface design help agents complete computer tasks better?. Whether a human or a model is on the other end, merging intent and screen-level action into one representation causes the same bottleneck.

There's also a reason the separation is getting harder to ignore. In conventional software, the interface is fixed and users learn it. In AI systems, the real context (prompt, history, retrieved data, hidden state) keeps shifting in ways users can't see How does AI context differ from conventional software context?. GUI controls are one of the few stable, inspectable handles a user has on that moving context. That's another reason to keep them as their own thing in the model instead of letting them dissolve into prompt text.

A caveat: only the IAI paper takes on the representation question directly. The other notes back it up from the design and agent-engineering side rather than formalizing it. The less obvious takeaway is that the split between "what you say" and "what you constrain" is the same split that makes GUI-operating agents work better. Interaction design and agent architecture are running into the same problem from opposite ends.