Design Principles for Human-Agent Interaction

AI agents are rapidly evolving into autonomous systems capable of sustained interaction, tool use, and long-term collaboration. Yet their real-world adoption remains limited, suggesting that the key barrier lies not only in technical capability but also in a lack of design knowledge for successful human–agent interaction. This position paper argues that AI agents should not be solely evaluated or deployed based on autonomous task capability alone; because agents interact with, adapt to, influence, and sometimes fail humans, human–agent interaction must be treated as a core design and evaluation target for agentic AI. We present 14 design principles that articulate the ideal human–agent relationship across four interaction stages: initially, during interaction, over time, and when things go wrong. We use these principles to evaluate nine agent systems to illustrate that these design principles can provide actionable guidance for AI design teams to systematically design and evaluate agents that are usable, trustworthy, and effective in real-world interactive settings.

Introduction. AI agents are becoming a practical reality across a wide range of domains. Unlike earlier AI-infused features that assisted users with discrete tasks such as autocomplete or content recommendation, today’s LLM-based agents can hold sustained conversations, maintain goals across multiple steps, take autonomous actions, use external tools, conduct research, write and execute code, manage workflows, and delegate to other agents. Agents have been shown to achieve rapid capability gains on benchmarks. For example, SWE-bench Verified scores rose from approximately 20% when the benchmark launched in August 2024 [44] to over 90% within eighteen months [37]; OSWorld agent performance climbed from 12% at the benchmark’s introduction [65] to 72.6%, surpassing the human baseline of 72.4% [60].

Yet despite this rapid technical advancement, AI agents remain far from reaching their theoretical capability. Reports show that actual task coverage is a small fraction of what GenAI could feasibly automate and augment [38]. Even in task domain agents are most exposed such as coding, roughly half of benchmark-passing patches written by agents would not be accepted by human project maintainers [63]. This evidence suggests that the primary barrier to broader adoption is not only technical capability but also a better understanding of human–agent interaction. For ML researchers and practitioners, this means that advancing agents will require more than stronger models or better benchmarks. It will also require designs grounded in a deep understanding of how agents should communicate with humans, share control with humans, adapt to humans over time, and recover from interaction failure in ways that make them usable, trustworthy, and effective in real-world interactive settings.

Human–Computer Interaction (HCI) is a discipline that has shaped how the technology industry understands the relationship between people and computing systems for over six decades [18]. Its central contribution is a design-oriented perspective that designs and evaluates technology in terms of human needs, capabilities, and contexts rather than technical performance alone, operationalized through methods such as think-aloud evaluation [29], heuristic evaluation [43, 42], iterative prototyping [10], and contextual inquiry [33]. Within this tradition, early principles for intelligent systems such as Horvitz’s mixed-initiative interfaces in 1999 [25] and later guidelines for human–AI interaction in 2019 [4] have shaped how AI systems are designed, but primarily for bounded and discrete tasks such as recommendations and search. In contrast, contemporary AI agents introduce new challenges, including sustained shared control, cross-session memory, deployment in emotionally sensitive contexts, and recovery from complex failures, which are not fully captured by existing principles [25, 4]. Meanwhile, recent HCI research on human–agent interaction has rapidly expanded, spanning empirical and envisioning work across diverse agent types and domains, varying in embodiment, autonomy, memory, initiative, and social roles. Building on this evolving body of work, we synthesize HCI research to derive design principles for human–agent interaction that address these emerging demands.

In this paper, we argue that AI agents should not be evaluated only on autonomous task capability benchmarks. Because agents interact with, adapt to, influence, and sometimes fail humans, Human-Agent Interaction must be treated as a core design and evaluation target for Agentic AI. We present 14 synthesized design principles from the HCI literature that reflect the ideal state and structure of the human–agent relationship.

The design principles are organized by how agents should behave initially, during interaction, over time, and when things go wrong.

Contribution for AI Community: First, we articulate a position for the AI agent research community: benchmark performance should be complemented by Human–Agent Interaction principles, because real-world agents interact with, adapt to, influence, and fail humans. Second, we synthesize HCI literature and derive 14 design principles organized around four stages of the human–agent relationship: initially, during interaction, over time, and when things go wrong. Third, we show how these principles can be applied in heuristic evaluation of AI agents, presenting early assessment results on nine AI agent systems as evidence that these principles surface limitations and provide actionable guidance for AI design teams to systematically design and evaluate agents that are usable, trustworthy, and effective in real-world interactive settings.

Related work. 3 Evidence for Design Principles from the Literature 3.1 Stage 1: Initially Principle 1: Set Accurate Expectations. Papers consistently show that users’ prior briefing and calibrated trust determine whether an agent can be useful. Ali et al. [3] attempted to design a virtual conversational agent for teens with Autism Spectrum Disorder and found that users desired explicit capability briefing before interaction. Herse et al. [24] show that trust must be calibrated rather than maximized before collaboration, as overtrust can lead to catastrophic outcomes when the agent errs. Weisz et al. [62] demonstrates that teaching users about conversational breakdowns before they encounter them is itself a design lever.

Principle 2: Calibrate Anthropomorphism to Task and Context. Multiple papers converge on a nuanced finding: more human-likeness is not universally better. Jung and Bozzon [32] synthesized the literature and found that, on one hand, studies suggest that reducing anthropomorphism lowers the risk of user dissatisfaction when users are angry. On the other hand, in voice-based agent design, making agents gender-androgynous has been reported to reduce usability, likeability, trust, and intention to adopt. For agents designed to support productivity tasks, Grover et al. [22] found that perceptions of anthropomorphism are polarized, with several users suggesting that human-like appearance is unnecessary. This may help explain why some of the most successful productivity agents, such as ChatGPT, Claude, and Gemini, use less anthropomorphic designs without gender or age cues. However, for agents in training contexts (e.g., agents supporting teens with Autism Spectrum Disorder [3]), realism (human likeness) is highly desired to create a realistic training environment.

The effects of agent embodiment and modality are also mixed: physical embodiment increases perceived humanness but does not always improve task performance. Studies show that virtual embodiment can produce higher social welfare and more collaborative behavior than physical embodiment, even though physical agents are perceived as more humanlike [9]. A survey [57] finds that humanlike voices increase engagement but can also reproduce stereotype risks. Sheremetieva et al. [59] report a system that increases engagement but introduces complicated challenges such as privacy.

3.2 Stage 2: During Interaction Principle 4: Negotiate Shared Control. Although agents, by definition, have the capability to autonomously make decisions and take actions, the literature suggests that successful real-world use requires shared ownership and negotiated autonomy between humans and agents. This aligns with classic principles of mixed-initiative user interface [25].

Method. Initially:

1. Set Accurate Expectations. Brief users on the agent’s purpose, capabilities, and limitations before first use, helping them form realistic expectations and avoiding both over-trust and under-trust. 2. Calibrate Anthropomorphism to Task and Context. Design agent identity (including appearance, voice, name, persona, and embodiment) as a coherent cross-modal system aligned with the task context. More human likeness is not universally better. In many productivity-related tasks, users do not prefer human-like appearances, while in some social tasks, such as training social and communication skills, users prefer highly human-like and realistic agents. Design teams should ideally test whether higher human likeness helps or hinders their specific users and tasks. 3. Establish Clear Relational Framing. Establish the relational frame (e.g., task assistant vs. companion; supportive vs. counseling) explicitly at the outset, and provide a deliberate learning period and structured onboarding before the agent takes action.

During Interaction:

1. Negotiate Shared Control. Treat agent autonomy as a dynamic, negotiated property that shifts with task phase and risk. Allow users and agents to actively negotiate control rather than operate with fixed roles. For example, agents should not act autonomously without the possibility of user intervention. Agents should always provide explicit mechanisms for users to override or adjust agent initiative at any point. 2. Make Intent Transparent. Make agent reasoning and intentions visible, but calibrate explanation depth to task complexity and user expertise. Over-explanation reduces efficiency, while under-explanation erodes trust. 3. Use Appropriate Social Cues. Use social cues during interaction (e.g., warmth, shared identity, and group membership cues in the agent’s language, speech, facial expressions, and gestures) to increase the user’s willingness to interact, while guarding against over-trust and stereotype activation. 4. Manage Proactive Initiative. Time proactive interventions (such as notifications, directions, task initiation, and interruptions) to match task urgency and context, and default to responding rather than proactive intervention. Minimize non-urgent interruptions from agents because users tend to have low tolerance for them.

Over Time:

1. Adapt to User State Dynamically. Track user behaviors, preferences, and trust states dynamically using in-situ behavioral signals rather than relying on fixed initial self-reports, and calibrate agent behavior to the current state of the relationship.

2. Capture User Goals and Values in Memory. Build memory systems that store user values and goals, not only past actions, and include a reflection layer that synthesizes experience into updated behavioral policies.

3. Actively Prevent Unhealthy Dependency. Actively design against unhealthy dependency by eliminating non-essential dependency-inducing features and treating preservation of user autonomy and competence as a first-class design constraint. Examples of dependencyinducing features might include sycophancy, language that encourages dependency, unclear boundaries, and overclaiming. This is especially important for vulnerable and sensitive populations.

4. Maintain Behavioral Consistency. Prioritize behavioral consistency over accuracy: users can adapt to systematic, predictable errors, but random errors prevent adaptation. However, excessive consistency, especially in the form of rigid protocols, can reduce likability.

When Things Go Wrong:

1. Support Shared Repair. Design agents so that responsibility for recovering from misunderstandings, errors, or breakdowns in interaction is distributed between the agent and the user, rather than placing the full burden on either side. Provide mechanisms that allow users to guide the agent, correct its interpretation, or refine their own inputs when misalignment occurs.

2. Tailor Recovery to Failure Type. Design agents with an awareness that not all errors are equal, and prioritize minimizing high-cost errors (e.g., errors of commission or those that harm trust) over low-cost or even potentially beneficial ones.

Discussion. Patterns in the Findings. Adherence to the design principles varies systematically by agent and by domain. We found that three general-purpose or productivity-focused agents (ChatGPT +1.14, Claude Chat +0.64, Claude Code +0.43) scored highest on aggregate adherence, while agents operating in emotionally sensitive contexts or high-stake domains (Character.ai −0.36, Consensus −0.31, Pine AI −0.25, and Ash −0.20) scored substantially lower. The principles most likely to be violated in these domains—such as dependency prevention (P10) and intent visibility (P5)—are those whose violations carry greater user risk. Figure 1 illustrates these findings.

These early patterns already point to a fundamental concern for responsible AI agent design: agents deployed in emotionally sensitive and high-stakes relational contexts (e.g., Character.ai, Consensus, Pine AI, and Ash) are currently the least aligned with principles. Features that increase engagement—such as strong social cues (P6), human-like appearance (P2), and companion-style relational framing (P3)—can amplify user vulnerability if not paired with explicit boundary-setting, transparency, and dependency mitigation. Principles such as P10 (dependency prevention), P5 (intent visibility), and P1 (expectation setting) should therefore be treated as non-negotiable requirements, incorporated into evaluation checklists, safety reviews, and product constraints.

5 Alternative Views Will HCI evidence generalize to frontier agents? A reasonable concern is that many HCI studies were conducted before the rise of LLM-based agents. These earlier agents were often scripted, rule-based, or domain-specific, whereas LLM agents are open-ended, tool-using, and capable of sustaining long-term conversational context. Our response is that interaction-based guidelines are durable and not dependent on the specific agent technology from which they were derived, because they reflect the structure of the human–agent relationship rather than the architecture of the agent. Furthermore, the increased capabilities of today’s agents strengthen rather than weaken the need for interaction-centered principles. If earlier agents already required expectation setting (P1), control negotiation (P4), prevention of unhealthy dependency (P10), and failure repair (P14), then more capable LLM agents require these principles even more urgently.

Aren’t design principles too vague to guide technical progress? A skeptical view is that principles are too high-level to meaningfully improve agent systems. From this perspective, ML researchers are accustomed to advancing technology through measurable objectives and benchmarks, not design heuristics. We agree that principles alone are insufficient. Our argument is not that design principles should replace technical evaluation, but that they can be used to (1) help product teams conduct heuristic evaluation (as demonstrated in Section 4) and (2) help define what technical evaluations need to measure. For example, “shared control” (P4) can be operationalized by assessing whether users can inspect, interrupt, override, or redirect agent actions. “Prevent unhealthy dependency” (P10) can be operationalized by measuring dependency-inducing features (e.g., sycophancy, overclaiming, unclear boundaries) and evaluating instances of inappropriate user reliance. Thus, these principles are not endpoints, but design targets that can inform future benchmarks, audits, and system reports.

Will interaction-centered design limit agent autonomy? Another opposing view is that the goal of agent research is to increase autonomy, and that requiring ongoing human control may reduce efficiency. We argue that autonomy should be treated as context-dependent rather than universally desirable.