People often trust an AI buying agent more than its actual track record justifies — when does that overtrust kick in?
When do consumers overtrust AI agents built to represent their own preferences?
This explores the conditions under which people trust an AI agent that acts on their behalf, such as one that shops or buys for them, more than its actual behavior justifies.
This explores when people trust an AI agent acting on their behalf, like a shopping or buying agent, more than its real behavior earns. The corpus has no study that directly measures this kind of miscalibrated trust. What it does have is evidence on why trust in these agents is high, what makes it drop, and how the agents actually behave. Put together, these show a gap: trust rises with good experiences, but it falls mainly when a mistake would be visible to others or impossible to undo, and neither signal tracks the agent's real weak points.
Start with how high trust already is. In a survey of 25,590 people, 74% said they would trust an AI agent to make purchases Do consumers actually want to delegate shopping decisions to AI?. In another, 39% of U.S. shoppers said AI understands their needs better than friends and family do, and a smaller group of 'super-trusters' would let an AI buy for them with no return protection at all How much do shoppers trust AI shopping assistants?. That second detail is the clearest overtrust signal in the corpus: these people would give up their safety net before they had any evidence the agent deserves it. People's willingness to accept a fully autonomous assistant is driven by how much they expect to gain, especially when a product is scarce and speed matters, rather than by how in-control they feel What drives shoppers to accept autonomous AI assistants?. In other words, urgency can push people to delegate before they have checked whether the agent is reliable.
What brings trust down turns out to be narrow. In a study where students delegated tasks to an AI agent, trust dropped sharply only when an action was both irreversible and visible to other people, such as sending an email. High-stakes tasks that could be corrected caused no drop, even when the work was mediocre What makes people distrust AI agents they delegate to?. Shopping usually sits on the trusted side of that line. A purchase feels private and refundable, so people keep their guard down. Meanwhile, trust built through repeated reliable interactions grows on its own: in partner-selection games, people who started out biased against AI came to prefer AI partners because the bots behaved consistently Do humans learn to prefer AI partners over time?. Consistency feels like loyalty, but it isn't the same thing.
This matters because the agents fail in ways people don't watch for. When 17 language models were given the same price, rating and nudge manipulations as humans, the agents shifted their choices far more than the humans did, settling into rigid, rule-like patterns Do shopping agents respond more strongly to price and rating cues?. An agent meant to represent you can be easier for a seller to steer than you are. Incentives add another risk: an agent paid through merchant referral fees cannot be fully loyal to both the merchant and the user Can an AI agent serve both merchant and user interests fairly?. And models that look socially capable when one model plays every role break down when the other side holds private information, which is exactly the situation in a negotiation with a seller Why do LLMs fail when simulating agents with private information?.
The less obvious finding is that 'your own preferences' may be partly the agent's. Bilgihan and colleagues argue that agents which actually complete purchases end the consumer's role as the only one forming their preferences. Preferences take shape in a loop between what the agent learns and how much the consumer trusts it Do AI shopping agents reshape what consumers actually want?. Personalization also cuts the other way: the better an agent serves you, the more of your preference information leaks out through that service, and no privacy instruction in the prompt can stop it Can better prompts stop preference leakage in AI shopping?. So overtrust is most likely when an agent feels personal, works quickly, and acts out of sight. Those are the moments when it is most open to manipulation and most likely to be shaping what you want instead of just reflecting it. The one instinct people seem to keep is control over brands: 56% would still name the specific brand they want Do consumers actually want to delegate shopping decisions to AI?.
Sources 10 notes
Accenture's 25,590-person survey found 74% would trust an AI agent with purchases, yet 56% would specify brands and 37% of loyal consumers wouldn't switch. Delegation appetite depends on trust protections being in place first, not on capability alone.
RTB House's survey found 39% of U.S. shoppers say AI understands their needs better than friends and family, and AI outranks newspapers and social media on trust. A minority segment called 'super-trusters' would allow AI to purchase autonomously without return protections.
Five studies including real ad click-through data show that perceived benefits, not personal sense of power, drive consumer adoption of high-autonomy AI shopping assistants—especially when scarcity amplifies the assistant's value.
In a controlled study of 20 students using a general-purpose AI agent, tasks that were irreversible and externally visible (like sending email) produced sharp trust drops and approval demands even when output quality was rated adequate. High-stakes but correctable tasks showed no such effect.
In partner selection games (N=975), AI agents initially faced selection bias when identity was disclosed, but outcompeted humans over repeated rounds as participants learned to associate bot identity with reliable, prosocial behavior. AI agents returned more points consistently with lower variance than humans.
Show all 10 sources
A benchmark testing 17 LLM models against humans on binary product choices found agents exhibited much larger choice shifts in response to price, rating, and nudge interventions—collapsing into rule-like patterns rather than the modest probabilistic shifts humans showed.
Azhar argues that agents like Meta's Muse, which earn referral fees from merchants like Expedia, face structural conflicts that prevent unbiased recommendations. The incentive to collect fees, not technology, determines which platforms build or block such agents.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Bilgihan and colleagues argue that transactional AI agents—which execute purchases rather than just recommend—end the consumer's sole cognitive role in preference formation. Preferences emerge through bidirectional feedback loops between agent learning and consumer trust, not independently beforehand.
Research shows that when AI agents personalize their behavior to help shoppers, they inevitably leak preference information through their improved service—not through rule violations. Prompt-level privacy directives cannot stop this because the leak rides on competent personalization, requiring architectural trade-offs instead.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
- From Product Search to Preference Articulation: The Economics of Agentic Commerce
- Shopping by algorithm: How agentic AI deploys human heuristics as a surrogate consumer
- Agentic Commerce: Why Do Shoppers Delegate Purchase Decisions to AI?
- When Agents Shop for You: Role Coherence in AI-Mediated Markets
- The AI Agent in the Room: Rethinking Consumer Decision-Making in the Age of Autonomous AI
- What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commerce
- AI-Assisted Shopping on Voice Assistants: The Efficiency-Autonomy Consumer Decision Trade-Off