About a quarter of an AI design tool's stated goals never get built — does splitting up the instructions explain why?
Does prompt decomposition explain the one-in-four unimplemented design rationales?
This explores whether the roughly 25% of design rationales that generative UI tools fail to build could come from how prompts get split into parts and processed, as opposed to some other weakness in the tools.
This explores whether the way a prompt gets broken into parts explains why generative UI tools leave about a quarter of their stated design rationales unbuilt. The short answer: the corpus doesn't test that cause directly. It does hold several findings that make prompt structure a plausible suspect, along with a few that point somewhere else. The headline number comes from a benchmark of 24 tasks across five tools. About 25% of design rationales went unimplemented, rising to 34% for functional requirements, and the tools recognized only half of the UX principles embedded in their prompts Do generative UI tools actually implement their stated design rationales?. That last detail matters. If a tool never registers a principle, the failure happens when the prompt is read, before any code is written.
There is real evidence that prompts have internal parts that get treated unequally. Practitioners working together consistently split prompts into four components: input context, system instructions, output constraints, and few-shot examples How do teams decompose prompts when collaborating on generative AI?. Compression research shows that models don't need every part equally. Instructions and questions are fragile, while demonstrations can be cut heavily with little loss Can prompts be compressed without losing model performance?. A design rationale is usually a 'why' buried inside an instruction, not an explicit output constraint. That makes it exactly the kind of content that could get diluted when a tool parses or restructures a long prompt. The six-dimension prompt-quality framework adds another angle: weaknesses in one dimension, such as logic or instruction clarity, spill over into the others Can we measure prompt quality independent of model outputs?.
The more interesting lateral clue comes from agent architecture. When a task is split up, it matters who does the splitting. Separating a 'decomposer' model from a 'solver' model improves accuracy, and the decomposing skill transfers across domains while the solving skill doesn't Does separating planning from execution improve reasoning accuracy?. Splitting work across parallel subagents does the opposite. Each one makes unwritten assumptions its peers never see, so their choices conflict Why do parallel subagents produce conflicting design decisions?. A design rationale is exactly that kind of unwritten, cross-cutting intent ('keep this calm and minimal', 'favor accessibility'). If a tool breaks a UI request into separately generated components, the rationale can easily get lost at the seams. So decomposition could hurt or help. It depends on whether the overall intent survives the split.
A competing explanation is that models copy the form of a request without its substance. Chain-of-thought research shows that models reproduce the shape of reasoning, and that logically invalid examples work nearly as well as valid ones Why does chain-of-thought reasoning fail in predictable ways? Does logical validity actually drive chain-of-thought gains?. Applied to UI generation, a tool could produce something that looks like it follows the brief without honoring the reasons behind it. That's 'design theater', and no change to prompt structure would fix it. Similarly, a model can score perfectly on metrics while its internal structure is broken Can models be smart without organized internal structure?, which may be why passing outputs can hide missing rationales.
The design idea most worth taking from this comes from Lam and colleagues' just-in-time objectives approach. It makes the AI's design decisions visible at every step instead of hiding them Does visible design structure keep HCI expertise necessary as AI advances?. If the one-in-four gap comes from rationales silently dropping out between steps, the fix may be to make the tool state which rationales it is acting on, so a human can spot the missing quarter. Restructuring the prompt alone may not be enough. Telling the decomposition explanation apart from the 'copies form, not substance' explanation would take an experiment the corpus doesn't yet contain.
Sources 10 notes
A benchmark of 24 tasks across five tools found roughly 25% of design rationales go unimplemented, rising to 34% for functional requirements. Tools recognized only half the UX principles embedded in prompts.
Observing 39 practitioners across 13 design sessions, researchers identified a recurring four-part structure: input context, system instructions, output constraints, and few-shot examples. Different roles engaged with each component differently, blurring traditional design-engineering boundaries.
LLMLingua achieves up to 20x prompt compression with only 1.5-point performance drops by allocating less compression to instructions and questions, more to demonstrations, then pruning tokens using a small model's perplexity scoring aligned to the target LLM.
Research identifies six evaluable dimensions—Communication, Cognition, Instruction, Logic, Hallucination, and Responsibility—with 20 sub-criteria based on Grice, cognitive load theory, and instructional design. Improvements in one dimension cascade to others, revealing prompt quality as a structured space rather than a flat checklist.
Modular architectures with separate decomposer and solver models outperform monolithic LLMs, with decomposition ability transferring across domains while solving ability does not. The separation prevents planning-execution interference and produces more generalizable skills.
Show all 10 sources
Walden Yan argues multi-agent parallelism is fundamentally fragile: subagents embed unwritten assumptions in every action, so agents working independently converge on conflicting choices. The fix is reverting to single-threaded linear agents where context remains continuous.
CoT guides models to pattern-match reasoning structure rather than perform genuine inference. This explains distribution-bounded failures, why structural coherence matters more than content correctness, and why performance optimizes against interpretability.
Illogical chain-of-thought exemplars matched valid CoT performance on BIG-Bench Hard, showing that structural properties—not logical validity—drive the gains. The model learns the form of reasoning, not genuine inference.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Lam and colleagues argue that just-in-time objectives architecture makes design decisions between AI and interfaces visible, ensuring HCI expertise remains critical. By surfacing generation-and-evaluation steps at every interaction rather than hiding them, the architecture structures ongoing design judgment into the system itself.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Tools
- Prototyping with Prompts: Emerging Approaches and Challenges in Generative AI Design for Collaborative Software Teams
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- CoT is Not True Reasoning, It Is Just a Tight Constraint to Imitate: A Theory Perspective
- When More is Less: Understanding Chain-of-Thought Length in LLMs
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- What does Generative UI mean for HCI Practice?
- Automatic Prompt Optimization with "Gradient Descent" and Beam Search