Why don't design recommendations transfer cleanly to AI companions?
This research explores why design guidance from adjacent fields like trustworthy AI fails to apply straightforwardly to AI companions. It asks whether ethics and UX can be treated separately or must be negotiated together for each design choice.
The paper's central finding is that design recommendations written for adjacent domains, such as trustworthy AI and relational AI systems, do not transfer cleanly to AI companions, and that the friction comes from ethical and UX considerations being "deeply intertwined" rather than separable. The abstract frames AI companions as conversational agents "designed to cultivate emotional connection rather than primarily to support humans in instrumental tasks." It says prior work offers possible design solutions but leaves open "what makes AI companion design difficult as a design problem." The authors' answer is that existing recommendations "often require context-sensitive application." The discussion gives the general form: "what optimizes user experience for one individual in one situation may create risks for a different user or in another context."
The paper reaches this through four phases (literature review, practitioner co-analysis, internal heuristic evaluation, external expert assessment) that produced nine design principle areas where applicability was tense. The discussion excerpt covers four of them, and each relates ethics to UX differently. Safety was "consistently viewed as ethically non-negotiable, yet still required context-sensitive implementation." Predictability and Consistency spanned both domains but was "primarily associated with flexibility for UX optimization," even though consistency itself "was viewed as ethically important." Engagement and Empathetic Response were framed as UX-oriented principles that should "generally be upheld," with participants asking for moderation when excessive engagement might lead to dependency or when empathetic responses could enable harmful behaviors. So there is no single rule such as "ethics constrains, UX optimizes." The relationship has to be worked out principle by principle.
Read against the nearest notes, this paper moves the contingency up to the level of design principles. Can attachment theory prevent parasocial harm in AI companions? treats safety as a boundary that can be built into a module. This paper agrees safety is non-negotiable but adds that how the boundary is applied still flexes with users and contexts. How should chatbot design vary by relationship duration? also makes design guidance depend on a variable, there the time horizon. Here the variable is user and situation. Does chatbot personalization build trust or expose privacy risks? has the same shape at the feature level: one move that improves the experience also raises the risk. The companion paper says this holds for engagement and empathy too, where the UX gain and the potential for dependency or harm are the same behavior.
The excerpt does not say how many practitioners or experts took part or who they were. It does not name the other five principle areas or list which recommendations from which adjacent domains were tested. It also does not report whether context-sensitive application resolved the tensions or improved anything for users. The findings are stated as what participants "viewed" and "emphasized," not as measured user outcomes. The implication at that strength is modest: the nine areas work as a map of where borrowed guidance needs a context check before it is applied to a companion. They are not verified design rules. Engagement and empathy in particular should be treated as principles with a moderation condition, not as quantities to maximize.
Inquiring lines that read this note 2
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Why does polished presentation create unearned authority in AI outputs? How should designers communicate what AI systems truly are and can do?Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can attachment theory prevent parasocial harm in AI companions?
Explores whether psychological frameworks from human relationships—particularly attachment theory—can establish safety boundaries that protect users from unhealthy emotional dependence on AI systems while maintaining therapeutic benefit.
proposes a safety boundary; this paper adds that safety, though non-negotiable, still needs context-sensitive implementation
-
How should chatbot design vary by relationship duration?
Do chatbots serving one-time users need different design than those supporting long-term relationships? This matters because applying the same design to all temporal profiles creates usability mismatches.
another case of companion design guidance depending on a variable, time horizon there and user and context here
-
Does chatbot personalization build trust or expose privacy risks?
Explores whether personalization features that increase user trust and social connection simultaneously heighten privacy concerns and create rising behavioral expectations over time.
same pattern of a UX gain carrying a matching risk, shown at the feature level
-
What makes leaving an AI companion so emotionally difficult?
This research explores why the emotional closeness users develop with AI companions makes it hard to disengage. Understanding this dynamic matters because it reveals a hidden cost of intimacy in AI design.
Evidence for: in AI companions, what users value is also what holds them, so ethical disengagement risk and intimacy UX are one design problem
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Assessing the Applicability of Existing Design Recommendations to AI Companion Design: A Multi-Method Study
- A Framework of User Experience Principles for Human-AI Agent Interaction in the Workplace
- AI Peers Exert Social Influence on Human Dishonesty in Groups
- Large Language Models for User Interest Journeys
- GenAI as a Power Persuader: How Professionals Get Persuasion Bombed When They Attempt to Validate LLMs
- Design Principles for Generative AI Applications
- The Addictive Intimacy of AI: Understanding User Disengagement from AI Companions and Why Some Relationships with AI Become Difficult to Leave
- Incoherent by Design? On the Moral Self-Consistency of LLMs
Original note title
ethical and UX considerations are intertwined in AI companion design — recommendations from adjacent domains need context-sensitive application