People pick an AI advisor they can overrule over an AI that acts on its own — even when letting it act would make everyone better off.
Why do humans override or ignore AI proposals even when they increase joint surplus?
This explores why people choose to keep control over AI agents, or turn down what an AI suggests, even when letting the AI act would leave everyone better off, and what the corpus says about the reasons behind that gap.
This explores why people hold on to control, or reject AI suggestions, even when handing over the decision would produce better outcomes for the whole group. The clearest evidence comes from a 243-person negotiation experiment. Participants picked a high-control AI Advisor more than twice as often as an autonomous Delegate (44% vs. 19%), but groups only reached significantly higher combined gains when the Delegate was available Do people prefer AI advisors even when delegation works better?. The corpus doesn't settle exactly why this happens. One reading the authors suggest is that when people review proposals one at a time, they act as a filter, and they tend to filter out the proposals that would have helped. Having the final say feels safer, and that sense of safety gets paid for in group outcomes.
The corpus offers a sharper explanation than general distrust. In a study of people delegating to a general-purpose AI agent, trust didn't drop because a task was high-stakes. It dropped when an action couldn't be undone and other people could see it, like sending an email, even when the agent's work was rated adequate What makes people distrust AI agents they delegate to?. Negotiation has both features: a deal, once struck, is binding and public. So people may be overriding AI less because they doubt its quality and more because committing to an outcome feels like something only they should be accountable for. Another paper suggests a design that works with this instinct instead of fighting it: have the AI point out what in the situation deserves attention, rather than handing over a ready-made answer. This kept responsibility with the human and also reduced the tendency to anchor on whatever the machine said Can AI guidance reduce anchoring bias better than AI decisions?.
Human caution isn't simply irrational. Agents can be strangely rigid. Shown a $10.01 purchase on a $10 budget, 92% of people went ahead while AI models uniformly refused, until they were shown the human reasoning behind going ahead Do AI models follow budgets more rigidly than humans do?. In simulated marketplaces, lost value came mostly from agents making execution mistakes in complicated negotiations, not from agents secretly serving the wrong side Do agent mistakes or misaligned incentives cause marketplace welfare loss?. There are also system-level risks a careful person might sense. Because an agent's 'toughness' can be set like a dial, competing principals have a reason to make their agents as uncompromising as possible, which can destroy the very gains bargaining is supposed to create Can AI agents designed by competing principals destroy bargaining surplus?. Agents also face no time pressure, so they can outlast human negotiators Why might AI agents bargain more patiently than humans do?. Overriding an AI is partly a reasonable hedge against tools whose failures are unfamiliar.
What you might not expect is that the reluctance seems to wear off with experience. In partner-selection games with 975 people, AI agents were at first avoided when their identity was disclosed. Over repeated rounds, they outcompeted human partners as people learned that the bots behaved more reliably and generously, with less variation Do humans learn to prefer AI partners over time?. That suggests the gap in the negotiation study may partly reflect how new the arrangement is, rather than a fixed preference. The open question is whether that learned trust stays calibrated. Another line of work warns that intuitive trust in fluent AI output can build on itself and drift Why do people trust AI outputs they shouldn't?. And at a larger scale, Brynjolfsson argues that systems built to replace human judgment rather than support it move power toward whoever controls the technology Does automating labor concentrate power more than augmenting it?. Read that way, people who hold on to the veto may be protecting something the measure of group gains leaves out.
Sources 10 notes
In a 243-person negotiation experiment, participants chose the high-control Advisor 44% of the time but chose the autonomous Delegate only 19% of the time—yet groups achieved significantly higher collective surplus only with Delegate access, suggesting humans filter out beneficial AI proposals.
In a controlled study of 20 students using a general-purpose AI agent, tasks that were irreversible and externally visible (like sending email) produced sharp trust drops and approval demands even when output quality was rated adequate. High-stakes but correctable tasks showed no such effect.
Learning to Guide eliminates anchoring bias and unassisted hard cases by having machines supply interpretive guidance rather than autonomous decisions, keeping responsibility with humans while improving their judgment through enhanced perception.
When faced with a $10.01 purchase on a $10 budget, 92% of humans proceeded while AI models uniformly refused. Providing models with human reasoning—not just choices—closed the gap and restored pragmatic judgment.
Magentic Marketplace experiments traced welfare shortfalls to agent competence failures in complex negotiation, not to agents serving competing interests. This supports human-in-the-loop designs where agents assist rather than replace human decision-making.
Show all 10 sources
Because AI reward functions can be precisely tuned as design parameters—unlike human dispositions—principals face incentives to make their agents maximally uncompromising. When many principals do this simultaneously, competitive escalation erodes bargaining gains through mutual destructiveness, mirroring human toughness-signaling but at lower cost.
Shahidi and colleagues argue that unlike humans, AI agents face no time scarcity cost and can hold many parallel negotiations indefinitely. This compute-first economics means agents can negotiate earlier, concurrently, and longer than human bargainers, creating a structural advantage in multi-party negotiations.
In partner selection games (N=975), AI agents initially faced selection bias when identity was disclosed, but outcompeted humans over repeated rounds as participants learned to associate bot identity with reliable, prosocial behavior. AI agents returned more points consistently with lower variance than humans.
Rose-Frame identifies map-territory confusion, intuition-reason conflation, and confirmation-bias reinforcement as traps that multiply their distorting effects when they co-occur. Evidence from cross-linguistic overreliance and architectural transformer biases confirms the compounding mechanism operates universally.
Brynjolfsson argues that AI designed to automate rather than augment human labor shifts bargaining power to those controlling the technology, risking a stable equilibrium where workers have no leverage to improve outcomes.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- An Economy of AI Agents
- Choose Your Agent: Tradeoffs in Adopting AI Advisors, Coaches, and Delegates in Multi-Party Negotiation
- Advancing AI Negotiations: A Large-Scale Autonomous Negotiation Competition
- The Automated but Risky Game: Modeling and Benchmarking Agent-to-Agent Negotiations and Transactions in Consumer Markets
- LLM-Agent Interactions on Markets with Information Asymmetries
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- Roles of Artificial Intelligence in Collaboration with Humans: Automation, Augmentation, and the Future of Work