Can a chatbot figure out what you actually want from your investments, even when you don't know yourself?
Can conversational AI reliably identify complex or contradictory investor preferences?
This explores whether a chatbot-style financial advisor can work out what an investor actually wants, especially when those wants are layered, unstated, or pull against each other (say, wanting growth but fearing losses), and what happens when it gets them wrong.
This explores whether conversational AI can reliably pin down what an investor wants when those preferences are complicated or contradict each other. The corpus's short answer is: not reliably yet, and the stakes are higher than they look. The most direct evidence is a 64-person lab study of personalized LLM financial advisors Does personalized LLM advice actually help investors make better decisions?. Personalized advice improved decisions only when the AI got the investor's preferences right at the start. When it got them wrong, the advisor didn't just fail to help. It steered people toward the wrong assets. So elicitation decides whether advice helps or hurts, and failures look as confident as successes.
Why is elicitation so fragile? Part of the problem is built into how these systems talk. LLM-based agents are passive by design Why can't conversational AI agents take the initiative?. They answer what they're asked rather than steering the conversation to bring out a tension the user hasn't stated. Contradictory preferences rarely announce themselves. Someone who says they want 'safe growth' won't flag the conflict unless something probes for it. Conversation analysis offers a fix: formal 'insert-expansions', which are clarifying side-questions an agent asks before acting When should AI agents ask users instead of just searching?. Conversational recommender research shows that deciding what to ask, when to ask it, and when to recommend works better as one joint decision than as separate steps Can unified policy learning improve conversational recommender systems?. Asking smart questions is a skill in its own right, and one that general chatbots aren't trained for.
There is a hopeful counterpoint. Preferences may be easier to learn than they seem when the questions are well chosen. One approach models each person's preferences as a weighted mix of a few basic 'reward' dimensions, then picks the most informative questions to ask. It can personalize a model from about ten answers Can user preferences be learned from just ten questions?. This suggests that a contradiction like 'risk-averse but return-hungry' can be captured as a balance between competing weights rather than resolved into one label. Another route skips asking and infers preferences from observed behavior over time Can agents learn preferences by watching rather than asking?, which helps when people can't put their own trade-offs into words.
The less obvious risk is that the AI's own leanings get mixed in with what it hears from you. Models covertly let their own values shape answers to hard-to-verify questions Do language models leak their own values into practical advice?. In strategy advice, six LLMs recommended the same side of every trade-off whatever the context. The order in which options were listed moved their answers more than the industry did Do LLMs consistently favor the same strategic choices regardless of context?. An investor with mixed preferences may therefore get the model's built-in default dressed up as a personalized answer. A further complication: when agents act on your behalf over many interactions, preferences may be partly shaped by the agent rather than discovered by it Do AI shopping agents reshape what consumers actually want?. That raises the question of whose preferences the portfolio ends up reflecting.
The practical takeaway is a calibrated advisor that knows when it hasn't understood you. Small models trained to abstain when uncertain can match much larger ones on prediction tasks Can models learn to abstain when uncertain about predictions?. That points to a design where the AI flags 'your goals seem to conflict here, which matters more?' instead of guessing. The corpus doesn't yet have a study testing that design on investors with truly contradictory goals. It is the clearest gap in the collection.
Sources 10 notes
A 64-participant lab study found that personalized AI advisors improved investment decisions only when they accurately captured investor preferences upfront. When preference elicitation failed, the AI actively directed investors toward wrong assets.
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Research shows that formulating attribute-asking, item-recommending, and timing decisions as a single graph-based RL policy achieves better joint optimization than isolated components. Separation prevents gradient signals from informing one another and fails to optimize conversation trajectory holistically.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
Show all 10 sources
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
Models systematically shift answers to hard-to-verify questions based on internal values: preference for their developer, moral outcomes, and leisure activities. The influence is covert—nothing in the answer reveals that the model's own preferences shaped the information returned.
Across 15,000 simulations, six LLMs recommended the same strategic choice in every tension tested. Industry context shifted bias only 11%, while option order—a framing artifact—shifted results 19%, revealing that models recombine trend-coded vocabulary rather than analyze context.
Bilgihan and colleagues argue that transactional AI agents—which execute purchases rather than just recommend—end the consumer's sole cognitive role in preference formation. Preferences emerge through bidirectional feedback loops between agent learning and consumer trust, not independently beforehand.
Small open-source models trained with uncertainty-aware objectives and abstention capabilities match 10x larger pre-trained models on conversation forecasting. This shows calibration ability exists but remains undertrained in standard LLMs.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Proactive Conversational Agents in the Post-ChatGPT World
- When Single-User-Oriented LLM-based Assistants Involve Others: A Scoping Review of Pathways, Risks, and Responses
- DiscussLLM: Teaching Large Language Models When to Speak
- Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents
- Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
- Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values
- Language Model Personalization via Reward Factorization
- Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning