Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant

Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of ‘LLM-modulo’ setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N= 248) of LLM

Introduction. Autonomous agents have been regarded as a research focus for artificial intelligence (AI) over the last century [1]. With the wish that autonomous agents can make our life better, many autonomous agents have been designed as virtual personal assistants [44]. These AI assistants (e.g., Siri) perform well (albeit imperfectly) in following user instructions to execute low-risk tasks like playing a song, reporting weather forecasts, or searching for an image to support everyday tasks. However, on tasks entailing potential risks (e.g., monetary payments or hiring an employee), humans hesitate to trust such AI systems due to loss aversion [90] and algorithmic aversion [11, 35, 66]. Only when users can obtain a sense of control by being able to modify the outcomes of imperfect AI can they overcome such algorithm aversion and be willing to collaborate with imperfect AI systems [12]. With the recent rise of large language models (LLMs) in natural language understanding and generation [108], researchers have started to analyze LLM-based agents and their applicability in a plethora of tasks [95, 103]. The term ‘LLM agent’ refers to an artificial entity based on LLMs that perceives its context, makes decisions, and then takes actions in response [103]. Compared to existing deep learning and LLM-based methods (e.g., chaining multiple LLMs [102]), LLM agents provide more flexibility in task solving and user interaction, which makes them suitable for daily assistance. This is primarily due to three reasons. First, with a planning module, LLM agents can generate a dynamic plan based on the tools provided [95, 103]. Such plans are typically defined in a logical structure — step-wise plans, which can be easily understood by humans. Second, with LLMs as a core control module, users can access and interact with external toolkits via a more natural interaction (i.e., conversation) with LLM agents [4, 108], reducing manual control efforts over function-specific tools. For example, LLM agents can complete time-consuming jobs like information seeking and information filtering (e.g., searching for a flight in itinerary planning) based on specific user needs. Third, the Markov decision process of LLM agents can generate a sequence of actions (i.e., using external toolkits)1 as output. Paired with an understanding of actions and necessary parameters for the interaction with the LLM agents, users can get involved in the real-time execution of tasks with LLM agents and fix potential problems while benefiting from task delegation [61]. Based on an intuitive framework for task delegation, Lubars et al. [61] found that user trust can play an important role in human delegation behaviors to AI systems. However, there is a relatively limited understanding of user trust development and calibration in collaboration with LLM agents. There is also a growing debate in the machine learning and AI research communities about whether LLMs can be truly considered as reasoning and planning agents [42]. With this in the backdrop, existing work on automated task completion has revealed that LLM agents can exhibit promising performance in handling complex tasks like playing games [99], answering complex questions [110], and in simulating social behavior [73]. However, such agents are still far from perfect. Due to the probabilistic nature of LLMs, there is much uncertainty in automating LLM agents for tasks with high risks attached. To avoid unintended or unexpected consequences, there is a need for user control over the real-time execution process. Through an empirical study of LLM planning capabilities, planning experts found that “LLMs’ ability to generate executable plans autonomously is rather limited” [91]. However, when combined with a sound planner in an ‘LLM-Modulo’ mode, “the LLM-generated plans can improve the search process for underlying sound planners” [91]. Humans can potentially be the ‘sound planners’ who can work in conjunction and optimize plans drafted by LLMs, which can then be executed by LLM agents. Such human-AI collaboration can reduce human efforts in generating a reliable plan from scratch. Attracted by the promise of LLM agents, there have been some early explorations [24] of adopting them in human-AI collaboration. However, the existing works have mainly analyzed how LLM agents can serve specific use cases (e.g., design creation [24]), and others have conducted structured interviews to obtain expert insights [107, 109]. It is still unknown how well LLM agents can work as general daily assistants to assist users in everyday tasks with varying stakes and how user trust and team performance are shaped by interacting with LLM agents. In our work, we address this research gap and adopt LLM agents to assist humans in everyday tasks by following a plan-then-execute workflow [96]. First, the LLM agent generates a step-wise plan formulated with a hierarchical structure.

Related work. Task Delegation. While humans prefer to play the leading role in human-AI collaboration [61], delegating to AI systems can bring benefits like cost-saving and higher efficiency. Apart from manual delegation decisions, it is common to apply automatic rules for human delegation (e.g., heuristics obtained from domain expertise or manually crafted rules [47]). Many user factors like trust [61], human expertise domain [17], and AI knowledge [75]) have a substantial impact on human delegation behaviors. Another relevant stream of recent research has explored AI delegation to humans [22, 65, 75]. Researchers have investigated the conditions under which AI systems should defer to a human decision maker, which may bring benefits of improved fairness [65], accuracy [70], and complementary teaming [31]. Compared to human delegation, AI delegation has been observed to achieve more consistent benefits in team performance [22, 34]. In collaboration with LLM agents, users need to determine when they should be involved in high-level planning and real-time execution. Such involvement decisions are similar to the delegation choices made by users. While task delegation is not the focus of our study, future work can explore this further.

Trust and reliance have been important research topics since human adoption of automation systems [15, 51]. Due to the widespread integration of AI systems in nearly all walks of society, there has been a growing interest in understanding user trust [68, 93] and reliance [16] on AI systems. User trust in the context of human-AI collaboration is typically operationalized as a subjective attitude toward AI systems/AI advice [51]. In comparison, user reliance on AI systems is based on user behaviors (e.g., adoption of AI advice and modification of AI outcomes). The two constructs have been shown to be highly related [50, 51]: for example, user trust can substantially affect user reliance [51]. However, they are intrinsically different and cannot be viewed as a direct reflection of each other [41]. Most existing work has, therefore, studied the two constructs separately in terms of subjective trust and objective reliance. Earlier work exploring human-AI trust primarily focused on the impact of different contextual factors surrounding user (e.g., risk perception [26]), task (e.g., task complexity [79]), and system (e.g., stated accuracy [105, 106]). Empirical studies have shown that most users tend to trust AI systems that are perceived to be highly accurate [105]. Such trust is vulnerable, as the AI system may provide an illusion of competence with persuasive technology (e.g., explanations [8, 27]) or overclaimed performance [105].

Method. In our study, we adopted plan-then-execute LLM Agents [96] as assistants to help users handle daily tasks, e.g., itinerary planning and currency transactions. Figure 1 illustrates how users collaborate with plan-then-execute LLM agents. First, the LLM agents will generate a step-wise plan based on a prompt specifying the plan format adopted from [36]. Then, users will make necessary edits to the plan based on the provided edit tools (will be further detailed in Section 3.2). After the user edit, we obtained the step-wise plan as outcomes of the planning stage. Next, the LLM agents will transform the step-wise plan into a sequence of action predictions, which will be served in a step-by-step manner. Users will join the real-time execution process and check whether they approve the current predicted action (i.e., blue card shown in Figure 1) or they would like to modify the current action prediction. The user involvement in execution stages will be introduced in Section 3.3. After the iterative execution of all steps, the task is solved. The evaluation of task performance is mainly based on the plan quality and execution accuracy of the action sequences.

While LLMs can generate high-quality plans, there is no guarantee of their correctness and their further impact on the execution of the plan. Thus, involving users in the planning stage and controlling the plan quality would be essential to ensure successful subsequent execution.

User-involved Planning. Figure 2 shows one screenshot of userinvolved planning in our study. At the top of the interface, we provide a task description along with three buttons: ‘Show Potential Actions’, ‘Plan Edit Instruction’, and ‘Add one step’. By clicking ‘Show Potential Actions’, we provide a prompt window to show concrete documentary descriptions of all potential actions (including action purpose and parameters) to be used in the execution stage. All instructions used in our tutorial are accessible with clicking the button ‘Plan Edit Instruction’. After users join the planning stage, an initial plan generated by LLM will be presented in the orange area. We allow users to edit the plan with following interactions:

User-involved Execution. Figure 3(a) presents a flow-chart to illustrate a primary step executed by the daily assistant (i.e., LLM agent). First, given one primary step, the daily assistant predicts an action based on a given list of prepared actions (i.e., pre-defined APIs in the backend). After users check the predicted action, they can choose from one of the following three buttons to respond. (1)‘Proceed’: It indicates users agree that the predicted action is correct. After clicking this button, the LLM agent moves forward to execute it and shows the execution result of this action. (2) ‘Feedback’: Users can give text feedback based on the message input area at the bottom of the conversational interface. This triggers another action prediction based on the current primary step and user feedback. Then, users are provided with the three options to proceed again. (3) ‘Specify Action’: Users can override the current action prediction with the manual specification of one action. If users choose this response, they are first asked to choose one action from the prepared action list and then fill in the parameters manually. The LLM agent directly executes the user-specified action. After one action is executed, if users are not satisfied with the results, they can choose to re-execute this step by providing text feedback (i.e., by clicking button ‘Give feedback and try again’), which works similarly to the ‘Feedback’ option. If users are satisfied with the execution results, they can click the ‘Next Step’ button and move to execute the next primary step. By iterating over this process through the step-wise plan, users can choose to either approve or get involved in modifying the execution outcomes in each step. All actions are predicted and executed in the backend (i.e., the respective API calls are triggered).

Discussion. Our experimental results show that user involvement in the planthen-execute workflow with LLM agents can help fix imperfect plans in planning and wrong action predictions in the step-by-step execution. However, user involvement does not ensure a consistently positive impact on calibrated trust and overall task performance across different tasks.

User Involvement Fails to Calibrate User Trust. Overall, user involvement in the planning and execution does not significantly impact user trust and calibrated trust in planning and execution outcomes. As Table 3 shows, user involvement in planning can harm plan quality in tasks with a high-quality initial plan, which may potentially cause worse task performance in the subsequent execution stage. Our experimental results do not support H1 or H3, which indicates user involvement does not necessarily help calibrate user trust in our study. Instead, with a task-specific correlation analysis (cf. Table 8), we found that the plan quality has a significant positive correlation with calibrated trust in both planning and execution outcomes. Combined with task-specific user trust and task-specific confidence, we can infer that users tend to trust the LLM agent overall. Such trust can be expected and calibrated in tasks with a high-quality plan. In contrast, users fail to calibrate their trust in the tasks where a low-quality plan is provided. A potential cause of such miscalibrated trust is the plausibility of plans generated by LLMs (i.e., plans that appear to be likely correct). In our study, all initial plans are formulated with a clear, logical structure, which covers most of the task requirements. At first glance, such highquality text pieces seem quite plausible and trustworthy. We also received some open text feedback such as, — “The plans look nice, I do not find any space for improvement” and “the planning stage of the LLM assistant was helpful and trustworthy.” Findings from recent work on LLM-assisted fact checking corroborate this, wherein authors found that convincing explanations provided by LLMs can cause over-reliance when LLMs are wrong [85].

User Involvement can Benefit Task Performance. User involvement in planning and execution can positively impact overall task performance, especially execution accuracy. As the results in Table 3 and Table 4 show, user involvement in planning can help address imperfect plans (e.g., task-1 with grammar error). Doing so further contributes to improvements in the execution accuracy. After controlling the plan quality, we found that user involvement in the execution can provide the best execution accuracy among most tasks considered in our study (cf. Table 5). Based on the failure analysis (Section 5.3.3), LLM agents can make mistakes in executing high-quality plans, which can be attributed to prediction errors (i.e., wrong action name or action parameters) and prediction failures (i.e., failure to provide valid action prediction). In practice with deployed LLM services, there is no reliability guarantee for the generated plan in planning or predicted actions in execution. User involvement can play an important role in the plan quality control and risky action control, ensuring that only correct and safe actions are executed to obtain desirable task outcomes.

Other Findings. We also found some user factors and perceptions that affect user trust and task performance. As seen in Table 7, nearly all covariates show a significant positive correlation with user trust in the AI system. Some of these covariates also impact user trust in the planning and execution outcomes. Overall, these findings indicate that users who are familiar with such systems tend to show higher user trust.

Conclusion. This work empirically studied human-AI collaboration based on plan-then-execute LLM agents. Adopting such LLM agents in various everyday scenarios, we analyzed the impact of user involvement in the planning and execution stages on user trust and overall task performance. We provide various interactions in each stage to help users fix imperfect plans and modify execution outcomes. Our results suggest that the LLM agents can provide plausible text plans to cover task requirements, which can be convincingly wrong. As a result, users develop uncalibrated trust in the planning and execution outcomes, and user involvement in the planning and execution stages fails to calibrate user trust (RQ1). We also found that the plan quality substantially affects the subsequent execution accuracy. Thus, when user involvement in planning can fix imperfect plans, the overall task performance (i.e., plan quality, accuracy of action sequence, and execution accuracy) gets improved. However, user involvement in planning can also harm task plan quality where the original plan is good to begin with. As a result, the LLM agents demonstrate worse task performance in these tasks. In contrast, user involvement in execution brings a more stable positive impact on task performance (RQ2). Our results suggest that plausible but wrong LLM outcomes can be detrimental to user trust calibration and overall task performance. We discussed the impact of convincingly wrong LLM outcomes and provided potential solutions and insights for future work. Furthermore, we synthesized key insights for better control and effective collaboration with planthen-execute LLM agents. We also shed light on opportunities to design flexible collaborative workflows with human oversight for effective collaboration with LLM agents. Our results indicate that user involvement in the LLM agent workflow can be important in ensuring reliable task outcomes. Future work can further investigate how to detect and handle plausible but imperfect LLM outcomes and design effective interventions to fix such problems.