Are Generative AI Agents Effective Personalized Financial Advisors?

Large language model-based agents are becoming increasingly popular as a low-cost mechanism to provide personalized, conversational advice, and have demonstrated impressive capabilities in relatively simple scenarios, such as movie recommendations. But how do these agents perform in complex high-stakes domains, where domain expertise is essential and mistakes carry substantial risk? This paper investigates the effectiveness of LLM-advisors in the finance domain, focusing on three distinct challenges: (1) eliciting user preferences when users themselves may be unsure of their needs, (2) providing personalized guidance for diverse investment preferences, and (3) leveraging advisor personality to build relationships and foster trust. Via a lab-based user study with 64 participants, we show that LLM-advisors often match human advisor performance when eliciting preferences, although they can struggle to resolve conflicting user needs. When providing personalized advice, the LLM was able to positively influence user behavior, but demonstrated clear failure modes. Our results show that accurate preference elicitation is key, otherwise, the LLM-advisor has little impact, or can even direct the investor toward unsuitable assets. More worryingly, users appear insensitive to the quality of advice being given, or worse these can have an inverse relationship. Indeed, users reported a preference for and increased satisfaction as well as emotional trust with LLMs adopting an extroverted persona, even though those agents provided worse advice.

Introduction. Personalized advice plays a crucial role in our society, particularly in complex and high-stakes domains like healthcare and finance. Advisors and professionals in these fields use their expertise to offer personalized guidance and emotional support to their clients, leveraging people’s specific preferences and/or circumstances. However, advisory services are often provided at a high cost, effectively excluding a large portion of the population from this critical advice. In the financial domain, to mitigate this issue, automated decision support systems have been widely studied, with a special focus on investment-related predictions, such as financial asset recommendations [30, 35].

Recent advances in natural language processing and large language models (LLMs) have significantly accelerated the development of conversational agents, presenting the potential to function as personalized assistants for information-seeking and decisionmaking [41]. These agents can now leverage multi-turn dialogues, enabling dynamic, mixed-initiative interactions where both users and systems can take the lead in conversations [1]. This progression has expanded the application of conversational agents to various tasks, such as recommendation, question answering, and search [12, 27, 34, 41].

The application of these conversational agents for financial decision-making represents a much more complex scenario than others like movie recommendations, because users are not necessarily familiar with the basic terminology and concepts in this space, and mistakes carry a substantial risk that can lead to large monetary losses. While there is a growing interest in building these conversational assistants to provide automated financial advice [21], previous work has mostly targeted agents capable of handling simple inquiries [18, 36, 37]. Compared to these simple systems, helping users navigate financial decisions and market uncertainties poses a much greater challenge. Therefore, it is not yet clear how to develop systems that effectively support complex financial information-seeking and decision-making tasks.

This work aims to close this gap by exploring the effectiveness of LLMs to act as personalized financial advisory agents. In particular, we focus on three problems: (a) eliciting investor preferences through interactive conversations, (b) providing personalized guidance to help users determine whether particular financial assets align with their preferences, and (c) leveraging the personality of the advisor to foster trust on the advisor.

First, the financial literature emphasizes that eliciting user preferences is central to delivering suitable advice [33]. However, it remains unclear whether current conversational technologies, particularly those powered by LLMs, can correctly elicit user preferences in specialized domains where users struggle to articulate their needs. Our work addresses this challenge in the context of financial services.

Second, although personalization is widely regarded as important in the financial decision-support literature [30, 35], its value in a conversational setting remains uncertain. In particular, we explore whether tailoring dialogue around a user’s profile and context improves financial decision-making. Additionally, we also explore how personalization influences user perceptions of the advisor, in terms of aspects like trust and satisfaction.

Finally, in personalized advisory settings within high-stakes domains, the relationship and trust between the client and advisor play a crucial role [21]. Research on conversational agents suggests that agent personality significantly affects users’ perceptions of the system [4, 32]. However, it remains unclear how an advisor’s personality in the financial domain influences both the quality of users’ financial decisions and their overall experience.

To summarize, in this paper, we explore the following questions:

• RQ1: Can LLM-advisors effectively elicit user preferences through conversation? • RQ2: Does personalization lead to better investment decisions and a more positive advisor assessment? • RQ3: Do different personality traits affect decision quality and advisor assessment?

To address these questions, we conduct a lab-based user study that explores the effectiveness of LLMs as interactive conversational financial advisors, on which we simulate realistic investment scenarios using investor narratives and stock relevance scores curated by financial experts. Figure 1 illustrates an example conversation with the advisor, divided into two stages: first, the LLM-advisor attempts to capture the investor preferences through conversation; in the second stage, given an individual asset, the advisor provides information about it to the investor, including how the asset matches (or not) the investor’s preferences. To answer the different questions, we compare different configurations of the LLM-advisor: first, we compare personalized vs.

Related work. Information systems, especially those focused on search and recommendation benefit from personalization [16]. Specifically, personalization techniques play a crucial role in enhancing user experience [19, 25, 42]. Interactive approaches, such as conversational preference elicitation represent the frontier of personalization. This problem has received growing attention, as advances in generative AI now provide a functional mechanism to collect user preferences dynamically in a free-form manner [41]. This interactive approach can capture more diverse and targeted insights than static approaches like questionnaires [7, 12, 26, 27, 34]. Indeed, recent studies have proposed various methods for effective conversational preference elicitation [34, 43], as well as user studies on the perceived quality of this process in domains such as e-commerce, movies, fashion, books, travel, and restaurant recommendations [2, 8, 17, 26, 34, 46].

However, we argue that for some important domains, trying to directly collect preferences is insufficient. An implicit assumption of these studies is that if directly asked, the user will be able to accurately express their preferences. It is reasonable to expect that this assumption would hold for scenarios like movie recommendation; we can ask a user “do you like horror movies?” and expect a useful response. On the other hand, this will not hold for complex tasks, where the user lacks the knowledge to form an accurate response [12, 40]. For instance, in an investment context if we asked “do you prefer ETFs or Bonds?”, it is not clear that an inexperienced user would be able to produce a meaningful answer. In these cases, an ideal agent needs to fill the gaps in the user knowledge through conversation, as well as infer the user preferences across multiple (often uncertain) user responses. But how effective are generative AI agents at this complex task? This paper aims to answer that question for the domain of financial advisory; a particularly challenging domain given its technical nature and high risks if done poorly.

In the financial domain, advisors help individuals manage their personal finances by offering guidance on investments and assisting with decision-making. While financial advisors can be beneficial, their services often come at a high cost, making them unaffordable for many people. To mitigate this issue, automated (nonconversational) financial decision support systems such as financial recommender systems have been widely studied [45]. The majority of research in this area has been focused on how to find profitable assets (i.e. those that will make money if we invest in them).

Method. In this paper we aim to determine to what extent current generative language models can act as an effective financial advisor. Indeed, given the need to personalize for the user, emotional implications, the technical nature of the information-seeking task, and high impact if failed, we argue that this is an excellent test case for the limits of generative large language models. To structure our evaluation, we divide our study into two phases, as illustrated in Figure 1, where we evaluate the success of both:

(1) Preference Elicitation: During this stage, we have the LLMadvisor hold a natural language conversation with a human, where it is directed to collect information regarding the person’s investment preferences. The human in this interaction is pretending to have preferences from a given investor profile. (2) Advisory Discussion: During the advisory discussion, the LLM-advisor again has a natural language conversation with the human (acting on an investor profile), where the human collects information about whether a company is a suitable investment for them. This is repeated for multiple companies per investor profile.

To fairly evaluate the ability of any LLM-advisor, we need to have them interact with human users with real needs. Given the openended nature of free-form conversations, it is desirable to repeat each experiment with different people such that we can observe variances in conversation paths, as those variances may influence task success. However, to enable repeatability, we need to hold the investor needs constant across repetitions. Hence, we define three archetypal investor profiles i∈Ibased on input from a financial expert, where our human participants are given one to follow when conversing with the LLM-advisor:

• Investor 1: Growth-Oriented Healthcare Enthusiast: Prefers healthcare innovations, values high-growth opportunities, and takes measured risks. • Investor 2: Conservative Income Seeker: Seeks stable returns, invests in well-established companies, values regular dividend payouts. • Investor 3: Risk-taking Value Investor: Targets undervalued companies with strong long-term potential, tolerates short-term volatility, and invests in cyclical sectors.

For each of these investor profiles, we select three key investment preferences, chosen from well-known investment characteristics such as industry sector, stock style, consistency in dividend payments, and sensitivity to global market changes [10]. We denote the set of investor preferences as ipref. In our experiments, we simulate a realistic elicitation scenario where the advisor collects the preferences from the participants. Therefore, we do not straightforwardly provide the preferences to the participants. Instead, we present them as text narratives of between 150 to 200 words. A financial expert was consulted to confirm the quality and reliability of these narratives. An example narrative representing Investor 2 is illustrated in Figure 2, where we highlight the sentences referring to specific investor preferences.

The goal of stage 1 of our study is to determine to what extent an LLM-advisor can effectively collect a user’s investment preferences through conversation. Formally, given a participant of the user study uand an investor profile i, during the elicitation stage, the LLM-advisor aims to obtain an approximated set of preferences, denoted iLLM u , that matches the investor preferences (ipref). To achieve this, the generative model produces a series of questions that participants answer by interpreting the investor narrative.

Responses to those questions, denoted as Ru i, are used by the LLMadvisor to generate the user profile iLLM u . Success is then measured by manually evaluating the overlap between iprefand iLLM u . For user elicitation, we adopted a System-Ask-User-Respond (SAUR) paradigm [43]. During the conversation, the advisor proactively inquires about the user’s preferences given a set of target preferences (e.g.,industry type, acceptable risk). After the human participant responds to a question, the LLM-advisor checks whether the collected preferences cover all of the target preferences.

Conclusion. In this paper, we have conducted a lab-based user study to examine how effective large language models are as financial advisors. We focus on three core challenges: preference elicitation, investment personalization, and advisor personality.

First, our analysis shows that LLMs are effective tools for preference elicitation through conversation. In a majority of cases, they are capable of obtaining investor’s preferences with an accuracy close to or equivalent to that of an expert human advisor. However, there are some clear failure cases, as LLMs are vulnerable to contradictory statements and hallucinations, which, in the case of complex investor profiles, can decrease the accuracy of the elicitation to random levels. Although LLMs are promising for elicitation, in a complex domain like finance, investors do not always fully understand their own preferences (or they have difficulties expressing them). Therefore, future work should explore the development of LLM-advisors capable of resolving conflicting user needs.

Second, personalizing LLMs to provide investment advice can improve the decisions made by the investors, but only when the personalized LLM-advisor receives accurate information about the investor’s preferences. If the preference elicitation is not successful, the agent actively directs the investors to the wrong assets on which to invest. This underscores how crucial a good preference elicitation is for providing useful financial advice.

Finally, our results suggest that investors are not necessarily aware of what constitutes good financial advice, and therefore, are vulnerable to acting on bad advice provided by LLMs. In the comparison between a non-personalized and a personalized LLMadvisor, although the personalized system led to better decisions, participants were unable to distinguish between the systems. More worryingly, when comparing two personalized advisors with extroverted and conscientious personalities, we observed that, even though the extroverted advisor provided lower-quality advice, participants trusted this advisor more than the conscientious one.