Do text-based GUI agents actually work in the real world?

ShowUI's framing critique is that the dominant GUI agent paradigm — language-based agents calling closed-source APIs with text-rich meta-information like HTML or accessibility tree — assumes oracle access that real-world deployment does not have. Users interact with interfaces visually through screenshots, without the underlying structural information that text-based agents depend on. The text-only approach is therefore architecturally limited regardless of model scale.

But GUI visual perception is not a problem natural-image MLLMs solve well. UI tasks need specialized capabilities — element grounding, action execution — rather than the conversational abilities multimodal chatbots are tuned for. ShowUI proposes three innovations addressing the resulting gaps.

UI-Guided Visual Token Selection treats screenshots as UI-connected graphs and adaptively identifies redundant relationships, using these as criteria for token selection during self-attention. This reduces compute by exploiting that screenshots are not natural images — large portions are visually redundant (background, repeated elements) and the connectivity structure of UI components encodes which tokens carry information.

Interleaved Vision-Language-Action Streaming unifies diverse needs within GUI tasks — managing visual-action history during navigation, pairing multi-turn query-action sequences per screenshot to enhance training efficiency. Treating vision, language, and action as a single interleaved stream is more flexible than the staged pipelines that dominate prior work.

Small-scale High-quality GUI Instruction Datasets result from careful curation and resampling against type imbalance — the data-side intervention that lets the architectural innovations actually train.

The implication is that GUI visual agents are not a special case of multimodal models — they are a domain where the visual prior, the action vocabulary, and the data distribution all need to be UI-shaped from the start. This is the strong end-to-end position that creates a tension with the perception-factoring camp: Why do vision-only GUI agents struggle with screen interpretation? and Can structured interfaces help language models control GUIs better? both keep the foundation MLLM general-purpose and add a structured perception layer. ShowUI argues the perception layer should be inside the model, made UI-shaped end-to-end. The two camps disagree on whether to factor perception out or train it in.