How should agents interact with humans across their relationship lifecycle?

A position paper argues that "AI agents should not be solely evaluated or deployed based on autonomous task capability alone; because agents interact with, adapt to, influence, and sometimes fail humans, human–agent interaction must be treated as a core design and evaluation target for agentic AI." The paper motivates this with a capability-adoption gap: SWE-bench Verified scores rose from approximately 20% when the benchmark launched in August 2024 to over 90% within eighteen months, and OSWorld agent performance climbed from 12% at the benchmark's introduction to 72.6%, surpassing the human baseline of 72.4% — yet "roughly half of benchmark-passing patches written by agents would not be accepted by human project maintainers." From this gap it synthesizes "14 design principles that articulate the ideal human–agent relationship across four interaction stages: initially, during interaction, over time, and when things go wrong."

The principles named in the excerpt, organized by stage: Initially — Set Accurate Expectations ("brief users on the agent's purpose, capabilities, and limitations before first use"), Calibrate Anthropomorphism to Task and Context ("more human likeness is not universally better"), and Establish Clear Relational Framing (make explicit whether the agent is "task assistant vs. companion; supportive vs. counseling"). During Interaction — Negotiate Shared Control ("agents should not act autonomously without the possibility of user intervention"), Make Intent Transparent ("calibrate explanation depth to task complexity and user expertise"), Use Appropriate Social Cues, and Manage Proactive Initiative ("default to responding rather than proactive intervention"). Over Time — Adapt to User State Dynamically, Capture User Goals and Values in Memory ("include a reflection layer that synthesizes experience into updated behavioral policies"), Actively Prevent Unhealthy Dependency (treating "preservation of user autonomy and competence as a first-class design constraint"), and Maintain Behavioral Consistency ("users can adapt to systematic, predictable errors, but random errors prevent adaptation"). When Things Go Wrong — Support Shared Repair and Tailor Recovery to Failure Type ("not all errors are equal"). Each principle is grounded in a cited HCI study rather than asserted from first principles.

This reproduces, almost exactly, the four-stage scaffold How should AI systems be designed across different interaction phases? used in 2019 for "AI-infused" systems generally — this paper keeps the same initially / during / over time / when wrong structure but respecifies the content for agentic systems specifically (negotiated control, values-in-memory, dependency prevention) rather than discrete-task interfaces like recommenders and search. It sits as a broader, HCI-literature-synthesized counterpart to What UX principles do workplace users want in AI agents?, which distills a narrower, workplace-specific principle set validated with business users rather than synthesized from published studies. The "Negotiate Shared Control" and "Make Intent Transparent" principles echo How can users verify what an AI agent actually plans to do? and overlap with the autonomy-as-design-choice framing in Can we design AI autonomy separately from capability?.

The excerpt presents this as a synthesis of prior HCI findings rather than new primary research, and the paper's own "Alternative Views" section concedes the obvious objections — that the underlying HCI evidence predates LLM agents, and that "principles are too high-level to meaningfully improve agent systems" — without resolving them through new data, only arguing the principles are "design targets that can inform future benchmarks, audits, and system reports" rather than endpoints. The excerpt also does not establish how the stated "14" principles map onto the full list beyond the dozen or so named and evidenced here, since the available text elides parts of the evidence section. The implication is that the framework is best used as a checklist for structuring heuristic evaluation and benchmark design, not as a validated predictor of adoption or trust outcomes.