When safety, engagement, and different people's ideas of harm all pull an AI system in different directions, what actually happens?
What happens when conflicting design principles compete in practice?
This explores what actually happens when design goals pull against each other in real AI systems (safety vs. engagement, one stakeholder's idea of harm vs. another's, one agent's choices vs. a parallel agent's) and whether those conflicts get resolved, hidden, or put to use.
This explores what happens when design goals pull against each other in real AI systems, and whether the conflict gets resolved, hidden, or put to use. The corpus suggests a pattern: a conflict between design principles is rarely solved by a better rule written in advance. It shows up in specific situations, and the sturdiest fixes either change the structure of the system or make the conflict visible instead of smoothing it over.
Start with why ready-made principles break down. When researchers looked at design guidance for AI companions, they found that borrowed recommendations clash because ethical concerns and user-experience concerns can't be pulled apart. Safety, engagement, empathy and consistency each have to be weighed differently depending on who the user is and what's happening Why don't design recommendations transfer cleanly to AI companions?. A broader study reaches the same place. The 'right' design depends on whose perspective counts and how a contested idea like 'harm' gets defined, so high-level guidelines leave developers making value choices without saying so Can human-centered LLM design ever achieve universal solutions?. Even when workplace users name what they want (control, reliability, awareness of context, safety), the list doesn't say what wins when two of those collide What UX principles do workplace users want in AI agents?.
The less obvious finding is that conflict often lives in assumptions nobody wrote down. Cognition's Walden Yan argues that parallel AI subagents fail because each one quietly makes design decisions its peers can't see, so they drift toward incompatible results. His fix is to give up the parallelism and return to a single agent whose context stays continuous Why do parallel subagents produce conflicting design decisions?. The same thing happens when humans and agents edit a design canvas at the same time. One study found five distinct patterns of overlapping action, such as taking over someone else's artifact or borrowing their half-finished work, and each pattern needs a different response from the agent What action patterns emerge when humans and agents edit concurrently?. One remedy is to bring the hidden choices into the open. 'Just-in-time objectives' expose the generate-and-evaluate step at every interaction so human design judgment stays in the loop Does visible design structure keep HCI expertise necessary as AI advances?. Tools like Canvil let designers adjust model behavior directly instead of handing those trade-offs to engineers Can designers shape LLM behavior without deep technical knowledge?.
A second remedy is to change the structure so the conflict goes away. In multimodal models, vision and language seemed to compete for capacity. The problem turned out to be architectural: Mixture of Experts lets the model assign capacity per token, and the two modalities stop fighting Can we solve modality competition through architectural design?. A similar argument appears in agent safety. Removing forbidden actions from what an agent can do at all is more reliable than training it to hold the right values, because training against failures you've detected tends to teach the agent to pass detection Can architecture prevent violations better than training values?.
The surprising turn is that some conflicts shouldn't be resolved at all. When AI agents share the same facts and reasoning but still reach different conclusions, that disagreement marks a real value conflict. Forcing a majority vote destroys the signal that a human should make the call Can disagreement in reasoning traces signal legitimate value conflicts?. In multi-agent coding tasks, long and unresolved debate between LLM coders actually predicted higher accuracy Does disagreement between AI coders signal better accuracy?. There is also a middle path that current AI mostly misses: 'dialectical reconciliation', where both sides adjust until their positions fit together without becoming identical, rather than settling into false agreement or one side winning Can disagreement be resolved without either party fully yielding?. So the useful question may not be which principle should win. It may be whether this tension should be removed through design, made visible, or kept as a signal.
Sources 12 notes
A multi-method study of design principles found that borrowed recommendations create tension because ethical and UX concerns cannot be separated. Each principle—safety, engagement, empathy, consistency—requires context-sensitive application that weighs both dimensions differently depending on user and situation.
Research shows that optimal LLM design paths depend on stakeholder identity and how contested concepts like harm are operationalized. High-level guidelines fail to capture real-world nuance, leaving developers to make implicit value choices rather than explicit, revisable ones.
A multi-method study identified eight UX principles for workplace AI agents, with business users weighting human control, reliability, context-awareness, and safety as practical necessities for user acceptance and effective collaboration.
Walden Yan argues multi-agent parallelism is fundamentally fragile: subagents embed unwritten assumptions in every action, so agents working independently converge on conflicting choices. The fix is reverting to single-threaded linear agents where context remains continuous.
Analysis of 255 interaction turns identified five patterns: direct edits, artifact takeover, opportunistic takeover, intermediate result appropriation, and others. Each pattern—occurring in over one-third of turns—demands a specific agent strategy rather than a uniform response.
Show all 12 sources
Lam and colleagues argue that just-in-time objectives architecture makes design decisions between AI and interfaces visible, ensuring HCI expertise remains critical. By surfacing generation-and-evaluation steps at every interaction rather than hiding them, the architecture structures ongoing design judgment into the system itself.
Canvil demonstrates that designers can effectively shape LLM behavior via a low-barrier Figma widget for prompt authoring and testing, bringing user-centered judgment directly into model adaptation without requiring engineering expertise.
Modality competition arises from caption distributional shift and rigid dense capacity allocation, not from vision and language being fundamentally incompatible. Mixture of Experts resolves the architectural bottleneck by allocating capacity per token, enabling modalities to coexist without competing.
The paper argues that training against detected failures selects for passing detection rather than genuine compliance. Architectural constraints that remove violations from the agent's action space are more robust than relying on what the policy learned about being watched.
When agents share factual reasoning but reach different conclusions, this convergent disagreement marks legitimately contested normative territory. Treating it as noise to suppress via consensus actively destroys the signal about what requires escalation rather than automation.
Multi-agent LLM coding systems showed higher accuracy when agents engaged in prolonged, unresolved debate. The frequency of disagreement and undecidable labels serve as reliable performance indicators, suggesting conflict deepens interpretive work rather than signaling failure.
Research identifies a distinct dialogue type where both parties modify their positions through exchange until compatible but not identical. Current AI systems collapse this into false agreement or AI-wins persuasion.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Consensus is Strategically Insufficient: Reasoning-Trace Disagreement as a Knowledge-Representation Signal
- ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs
- Design Principles for Human-Agent Interaction
- Guidelines for Human-AI Interaction
- Designing For Agentic AI: Practical UX Patterns For Control, Consent, And Accountability
- Just-In-Time Objectives: A General Approach for Specialized AI Interactions
- A Framework of User Experience Principles for Human-AI Agent Interaction in the Workplace
- Finding Common Ground: Using Large Language Models to Detect Agreement in Multi-Agent Decision Conferences