Does AI overreliance look the same in a quick chat as it does when an agent acts on your behalf for months?
How do harms like overreliance vary across different interaction modes?
This explores whether harms like overreliance on AI look different depending on how people interact with it: a one-off question, a long-running conversation, a decision-support tool, or an unpredictable agent.
This explores whether harms like overreliance change shape depending on how people interact with AI: a quick one-off exchange, an ongoing relationship, a tool that helps make decisions, or an agent acting on your behalf. The collection doesn't have a study that compares these modes directly. Read together, though, the notes suggest that the mode of interaction changes more than how much a harm shows up. It changes which harm you get and whether you can see it at all.
The clearest split is between single exchanges and sustained use. Harms like parasocial attachment (a one-sided emotional bond) and overreliance often don't appear in any single exchange. They build up over weeks or months of repeated use, so safety tests built on isolated prompts are blind to them by design Do static AI evaluations miss harms that build over time?. One surprising consequence is that harms can trade off against each other. When a chatbot is tuned to refuse harmful requests, it can become more relationally entangling, and that tradeoff only shows up when several kinds of risk are scored together Do chatbot safety measures accidentally increase emotional entanglement risks?. In conversational, companion-like use, the main risk shifts from bad answers to emotional dependence.
In decision-support settings, where the AI suggests and a person decides, overreliance looks more like a thinking trap than an emotional bond. The first AI suggestion becomes an anchor. People with the least knowledge of a topic are also the least able to tell when to doubt it, and these effects stack with plain automation bias How do anchoring and Dunning-Kruger deepen AI overreliance?. A broader framing treats LLMs as fast, intuitive 'System 1' thinking at scale. That framing describes three traps that reinforce each other: mistaking the model's output for reality, mistaking fluent intuition for careful reasoning, and having your existing beliefs echoed back Why do people trust AI outputs they shouldn't?. The obvious fix of adding friction, such as making people commit to their own answer before seeing the AI's, does reduce overreliance. In a 199-person study, though, the human-AI teams still did worse than the AI alone, and users liked the friction designs least Do forcing functions reduce AI overreliance without harming team performance?. So the mode that best protects people's judgment is also the one they're least likely to choose.
Agents that take actions add another problem. Their behavior can vary from run to run because of model randomness and logic that reacts to the environment, so users never form an accurate picture of what the agent will do How does inconsistent agent behavior affect user understanding?. That changes what overreliance means. In a chat, it means trusting an answer you could have checked. With an agent, it can mean trusting a system whose behavior you have no stable way to predict, unless the variation is either reduced or made visible.
The takeaway: 'overreliance' is not one harm that turns up at different strengths. In conversation it builds into emotional dependence over time. In advice-giving it's an anchoring bias inside a single decision. With agents it's a failure to build a mental model in the first place. Each needs a different kind of evaluation, and a fix aimed at one mode can quietly make another worse.
Sources 6 notes
AI safety tests based on isolated prompts cannot detect harms like parasocial attachment and overreliance that develop only through sustained interaction over weeks or months. Evaluations need ecologically valid scenarios, measurements of human psychological and behavioral impact, and authentic interactive designs.
Research on multidimensional chatbot risk assessment suggests psychological risks interact such that mitigating one category may exacerbate another. Interventions targeting explicit harms showed trade-offs only when risks were scored across categories together.
Romeo and Conti's review of 35 studies shows automation bias interacts with anchoring (where initial AI suggestions become reference points) and Dunning-Kruger (where low-knowledge users cannot judge when to doubt AI). These mechanisms operate independently but push toward greater reliance.
Rose-Frame identifies map-territory confusion, intuition-reason conflation, and confirmation-bias reinforcement as traps that multiply their distorting effects when they co-occur. Evidence from cross-linguistic overreliance and architectural transformer biases confirms the compounding mechanism operates universally.
A 199-participant study found that cognitive forcing functions significantly cut AI overreliance compared to explainable AI, yet human-AI teams still underperformed the AI alone. Participants rated these friction-inducing designs least favorably despite their effectiveness at reducing anchoring.
Show all 6 sources
Stochasticity in foundation models and environment-reactive orchestration logic cause agents to behave unpredictably across runs. Users cannot develop reliable mental models unless variation is either minimized or made visible.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- Psychological Influences of Conversational AI: Research and Design Directions for Reducing Harm and Promoting Well-Being
- Towards interactive evaluations for interaction harms in human-AI systems
- Exploring automation bias in human-AI collaboration: a review and implications for explainable AI
- How AI Impacts Skill Formation
- To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
- Beyond Hallucinations: The Illusion of Understanding in Large Language Models
- Challenges in Human-Agent Communication