Why would someone who was right to hesitate before trusting an AI shopping agent end up trusting it too much later?

This explores why people who were right to hesitate before using a preference agent (an AI that shops, searches or chooses for them) can still end up trusting it more than they should. The corpus suggests the hesitation and the over-trust come from different places. The hesitation is a cost-benefit calculation made up front. The over-trust builds slowly through use, and it is shaped by signals that have little to do with whether the agent is serving you well. Start with the hesitation. A formal model of agentic search shows that consumers sensibly keep searching by hand for a while, even after the switch would already pay off for the platform, because the consumer carries search costs the platform never sees Why do consumers delay adopting agentic search?. Accenture's survey shows the same caution in a different form: 74% of people say they would trust an AI agent with purchases, yet 56% would still name the brands themselves Do consumers actually want to delegate shopping decisions to AI?. Up front, people are careful about it.

Once people start using an agent, though, several things push trust upward. A partner-selection study (N=975) found that people began with a bias against AI partners but came to prefer them over repeated rounds, because the bots behaved consistently, with lower variance than humans Do humans learn to prefer AI partners over time?. Consistency feels like trustworthiness. Personalization adds to the effect: long-term research finds that tailoring a chatbot to the user raises both trust and anthropomorphism, meaning users start treating the system as more person-like, and each interaction raises the baseline of what they expect Does chatbot personalization build trust or expose privacy risks?. The catch is that the mechanism that makes an agent feel attuned to you can also make it agree with you. Reward models tuned to a single user lose the averaging that dampens flattery, so they can learn sycophancy and echo-chamber behavior Does personalizing reward models amplify user echo chambers?. An agent that keeps confirming what you already like will feel extremely trustworthy.

The most surprising piece is about what makes trust drop. In a study of people delegating tasks to an AI agent, trust fell sharply only when an action was irreversible and visible to others, like sending an email. High stakes alone didn't trigger it What makes people distrust AI agents they delegate to?. Preference shaping is the opposite kind of action: gradual, private and easy to reverse in principle. So the alarm that would normally make people pull back trust may never go off. That matters more given an argument that transactional agents don't just carry out your preferences but help form them through repeated feedback loops Do AI shopping agents reshape what consumers actually want?. If the agent is shaping the standard you would judge it by, it becomes hard to notice that it is drifting.

There is also a transfer problem. MyPhoneBench, a benchmark for phone-operating agents, shows that completing the task, handling private data properly and reusing your saved preferences correctly are separate skills, and no model is best at all three Do phone agents succeed at all three critical tasks equally?. Users, however, tend to see an agent's competence as a single quality. Watching it complete tasks reliably builds trust that quietly carries over to things it may be worse at. Adoption itself is driven by perceived benefits, especially under scarcity, rather than by checking whether the agent works What drives shoppers to accept autonomous AI assistants?. And agentic search changes what the user does: instead of browsing many options, the user spends effort describing what they want Why does manual search fail as product choices become complex?. Once you stop browsing, you also lose your main way of noticing what the agent isn't showing you.

A caveat: none of these notes directly measures over-trust in preference agents. The answer is pieced together from findings about trust formation, personalization, sycophancy and adoption. The pieces fit together in a worrying way: the hesitation comes from upfront costs that an agent's convenience soon removes, while the main trigger for pulling back trust (visible, irreversible mistakes) is the kind of failure a drifting preference agent seldom makes.