Does personalized LLM advice actually help investors make better decisions?

In the same 64-participant lab study, a personalized LLM-advisor was compared against a non-personalized one on investment advisory quality. The paper reports that "personalizing LLMs to provide investment advice can improve the decisions made by the investors, but only when the personalized LLM-advisor receives accurate information about the investor's preferences." When elicitation failed, the effect reversed: "If the preference elicitation is not successful, the agent actively directs the investors to the wrong assets on which to invest."

The mechanism traces to the study's two-stage design: in stage one, the LLM-advisor runs a System-Ask-User-Respond conversation to build an approximated preference set from a human investor narrative; in stage two, it uses that approximated profile to discuss whether specific assets suit the investor. The paper finds LLM-advisors "match human advisor performance when eliciting preferences" in most cases, but are "vulnerable to contradictory statements and hallucinations" on complex investor profiles, where elicitation accuracy "can decrease... to random levels" — and because stage two's advice depends entirely on stage one's output, elicitation failures propagate directly into misdirected advice.

This gives a mechanism for why personalization doesn't reliably help: How do personalization granularity levels trade precision against scalability? frames user-level personalization as highest-precision but constrained by data sparsity; this study shows that even with a dedicated live elicitation stage rather than sparse historical data, user-level precision still depends on a fragile upstream step that can fail on complex or contradictory profiles. It also qualifies Does chatbot personalization build trust or expose privacy risks?: the benefit side of that dual dynamic is here shown to be conditional, not automatic — personalization can actively harm decision quality when the underlying elicitation is wrong, not merely fail to help.

The excerpt does not quantify what fraction of elicitation attempts failed to the point of misdirection, nor which investor profile(s) drove accuracy down to "random levels," beyond citing complexity and contradictory statements as triggers; it also does not test whether structured elicitation techniques would close the gap against the free-form conversational approach used here. At the strength the study supports, the implication is that personalized financial-advice products built on conversational elicitation carry a specific risk: the personalization layer meant to improve advice can instead degrade it, and the user has no reliable way to detect which case applies, since the companion finding in Do users trust extroverted AI advisors despite worse advice? shows that users' trust doesn't track advice quality anyway.