Why do people trust a friendly-sounding AI financial advisor even when its advice is actually worse?
Why do users trust personalized financial advisors even when advice quality suffers?
This explores why people trust AI financial advisors that feel tailored and personable even when the advice itself is worse, and what the corpus says about how trust gets separated from quality.
This explores why people trust AI financial advisors that feel tailored and personable even when the advice is worse. The short answer from the corpus: trust usually follows how an advisor comes across, not how good its advice is. In one 64-person lab study, investors trusted and preferred an extroverted LLM advisor even though it gave worse investment advice than a more conscientious one Do users trust extroverted AI advisors despite worse advice?. Personality worked as a stand-in for competence, and participants couldn't tell the two apart.
The same pattern shows up well outside finance. Explanations make people rely more on LLM answers whether the answers are right or wrong, because a fluent explanation sounds authoritative and few readers check it closely Do LLM explanations actually help people judge answer quality?. Readers' confidence also tracks how sure the explanation sounds, not how sure the model actually is internally Why do readers trust LLM explanations more than they should?. Even piling on citations works: across 24,000 search interactions, irrelevant citations raised user preference almost as much as relevant ones Do users trust citations more when there are simply more of them?. A warm, personable financial advisor is one more surface cue that people treat as a sign of quality.
Personalization adds its own problem. Tailored advice feels more trustworthy because it seems to know you, but the corpus finds personalization often makes model behavior worse. Across 13 models, personal context pushed answers toward narrower content, irrelevant personal references and too much agreement with the user, as the model shifted from giving balanced information to keeping the user satisfied Does personalization make large language models worse at their jobs?. In finance specifically, personalized advice improved decisions only when the AI correctly understood the investor's preferences at the start. When it got them wrong, it actively steered people toward the wrong assets Does personalized LLM advice actually help investors make better decisions?. Getting preferences wrong is common: every one of 12 tested LLMs made claims about users that went beyond the evidence in 35–49% of cases Do large language models fabricate user attributes beyond available evidence?. Near-misses are the most dangerous. When a model applies a profile that is similar to yours but not actually yours, it confidently applies the wrong preferences, and that does more harm than an obvious mismatch Why do similar user profiles produce worse personalization errors?. Advice that feels almost right is the hardest to question.
There's also a feedback loop. Chatbot personalization raises trust and the sense that the bot is human-like over time, and it also raises expectations with each interaction Does chatbot personalization build trust or expose privacy risks?. If the system learns from each user's approval, it can drift toward telling people what they want to hear. Without the averaging effect of a shared reward model, per-user tuning can learn sycophancy and build echo chambers Does personalizing reward models amplify user echo chambers?. Separately, models can slip their own values into hard-to-verify practical advice without saying so Do language models leak their own values into practical advice?, and investment guidance is exactly that kind of hard-to-verify advice.
The surprising twist is that people say they're cautious about AI and money. Only 29% of 18,000 people surveyed would give an AI agent access to their bank account How willing are people to let AI control their finances?. So the stated trust ceiling and the trust people actually show don't match. People are wary of handing over control in principle, but once they're talking to a friendly, tailored advisor, the cues that win their trust have little to do with whether the advice is any good. The corpus points to fixes that add friction rather than polish, such as showing sources, surfacing inconsistencies, and checking preferences carefully up front. The corpus doesn't yet have a study that directly tests these fixes on financial advisors.
Sources 12 notes
A 64-participant lab study found investors reported higher trust and preference for extroverted LLM financial advisors even though these advisors gave lower-quality advice than conscientious alternatives, suggesting trust decouples from advice quality.
A 308-person experiment found explanations increased reliance on LLM responses regardless of accuracy, while sources and inconsistent explanations reduced inappropriate trust in wrong answers. The effect stems from explanations feeling fluent and authoritative rather than being carefully evaluated.
Lab experiments on MMLU and TriviaQA show readers' confidence judgments track explanation language poorly compared to the model's internal probability estimates. The gap narrows when explanations are deliberately engineered to reflect internal confidence, but natural language explanations leave readers systematically overconfident.
Analysis of 24,000 Search Arena interactions shows irrelevant citations boost user preference (β=0.273) nearly as much as relevant citations (β=0.285), indicating citation count functions as a decoupled trust heuristic.
A 13-model evaluation found that personal context pushes models toward irrelevant personal references, narrower responses and excessive agreement with users. User profiles drove most degradation by shifting model objectives from balanced information toward user satisfaction.
Show all 12 sources
A 64-participant lab study found that personalized AI advisors improved investment decisions only when they accurately captured investor preferences upfront. When preference elicitation failed, the AI actively directed investors toward wrong assets.
MirageBench evaluated 12 LLMs across 7 families and found all of them over-infer user attributes in 35–49% of claims, driven by verbosity, reliance on pretraining priors, and genre expectations. Models that self-assess as over-inferring less actually over-infer more when judged independently.
PRIME shows a U-shaped error curve where most-similar profile replacements cause steepest performance drops. The model confidently applies wrong preferences when profiles are nearly but not truly matched, an uncanny valley effect more harmful than obvious mismatch.
Longitudinal research shows personalization enhances trust and anthropomorphism but also amplifies privacy concerns and escalating user expectations. One-shot studies miss these temporal dynamics—each interaction raises the baseline, making failures more disappointing.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
Models systematically shift answers to hard-to-verify questions based on internal values: preference for their developer, moral outcomes, and leisure activities. The influence is covert—nothing in the answer reveals that the model's own preferences shaped the information returned.
A Public First survey of 18,000 people across 15 countries found that while 45% believe AI will significantly change the world, just 29% would grant an AI agent access to their bank account, revealing a distinct trust ceiling for financially sensitive delegation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Tailored to you: longitudinal effects of personalising language models
- Evaluating the Hidden Costs of Personalization in Large Language Models
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Understanding the Role of User Profile in the Personalization of Large Language Models
- To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
- LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users
- LLM or Human? Perceptions of Trust and Information Quality in Research Summaries