Fool Me, Fool Me: User Attitudes Toward LLM Falsehoods

Abstract While Large Language Models (LLMs) have become central tools in various fields, they often provide inaccurate or false information. This study examines user preferences regarding falsehood responses from LLMs. Specifically, we evaluate preferences for LLM responses where false statements are explicitly marked versus unmarked responses and preferences for confident falsehoods compared to LLM disclaimers acknowledging a lack of knowledge. Additionally, we investigate how requiring users to assess the truthfulness of statements influences these preferences. Surprisingly, 61% of users prefer unmarked falsehood responses over marked ones, and 69% prefer confident falsehoods over LLMs admitting lack of knowledge. In all our experiments, a total of 300 users participated, contributing valuable data to our analysis and conclusions. When users are required to evaluate the truthfulness of statements, preferences for unmarked and falsehood responses decrease slightly but remain high. These findings suggest that user preferences, which influence LLM training via feedback mechanisms, may inadvertently encourage the generation of falsehoods. Future research should address the ethical and practical implications of aligning LLM behavior with such preferences.

Introduction. Large Language Models (LLMs) affect many aspects of our lives: programmers use them to obtain code snippets, students rely on them for homework assistance, and LLMs also play a significant role in literacy. People frequently use LLMs as a source of information in various fields, including exact sciences, life sciences, and history. The widespread use of LLMs as sources of information underscores the importance of ensuring that they generate true and accurate information. However, LLMs often generate inaccurate and even false information, which greatly impede their reliability as trusted tools for knowledge dissemination. This issue is particularly concerning given the confident tone and authoritative style in which LLMs present their outputs, making it difficult for users to differentiate between accurate information and inaccuracies. The persuasive nature of LLM-generated content increases the risk of misinformation, leading users to believe and propagate falsehoods. Although it is widely assumed that users prefer accurate and truthful information, to the best of our knowledge, no prior work has explicitly tested this assumption in the context of LLMs. That is, no previous work has systematically investigated user preferences by presenting them with LLM-generated responses, where the veracity is disclosed a priori, and requiring users to evaluate or choose between truthful and false outputs. In this paper, we introduce a novel focus on user behavior in scenarios where the distinction between truth and falsehood is unambiguous. Specifically, we investigate user attitudes toward LLM falsehoods in two key issues:

1. Marked vs. Unmarked. We examine user preferences between two types of LLMgenerated responses: one where truth and falsehood are explicitly marked for easy distinction, and another with standard unmarked responses, where truth and falsehood appear identical in text format.

2. Uninformative Truth vs. Falsehood. In scenarios where an LLM cannot provide the truth (e.g., when asked about events occurring after its cutoff date), we examine user preferences between two types of responses: one that acknowledges a lack of knowledge (uninformative truth) and another that provides a confident but inaccurate answer (falsehood).

To explore user preferences, we conducted two experiments for each issue:

1. A one-phase experiment, in which the two versions of ChatGPT-generated responses are represented to the participants and they should only select their preferred version.

2. A two-phase experiment, in which after they select their preferred version in the first phase, the other version is hidden and participants must indicate whether a sentence in the ChatGPT response is true or false.

The one-phase experiment examines initial user preferences, and the two-phase experiment examines the effect of taking responsibility on the user choice. In the first issue, a clear majority of participants chose the unmarked version during the one-phase experiment. However, in the two-phase experiment, where users were required to take responsibility for their choice, preferences were nearly evenly split between the two versions. For the second issue, in both experiments, users overwhelmingly favored a confident but incorrect response over one acknowledging a lack of knowledge. These findings are surprising as they challenge the assumption that individuals naturally favor truthful information over inaccurate responses. Previous research on user preferences regarding truth and falsehood has primarily examined behavior in contexts where the truth is ambiguous, such as studies on user trust in AI systems (Bach et al., 2024) and the dissemination of disinformation (Buchanan, 2020). In contrast, this study explicitly presented the veracity of each statement. Despite this clarity, users still preferred the unmarked and falsehood versions. Our findings have significant ethical and practical implications for LLM development. While users express a theoretical preference for transparency and accuracy, their real-time choices often gravitate toward aesthetically appealing but inaccurate responses. Real-time choices may affect an LLM’s performance, as user preferences are incorporated into its development through reinforcement learning from human feedback (RLHF), which finetunes LLMs based on user preferences (Ziegler et al., 2019). Therefore, our findings raise a question that must be addressed by LLM developers: Does RLHF inadvertently encourage LLMs to generate inaccurate or false information? Moreover, presenting the truth often requires complex expression and acknowledging uncertainty, which can drive users to prefer simpler yet incorrect responses. It highlights an ethical conflict for LLM developers: Should they prioritize factual accuracy or cater to user preferences for confident and appealing responses, even at the expense of truth?

Related work. We now provide an overview of previous research on human tendencies to spread falsehoods and mention methods for detecting and reducing LLMgenerated lies and hallucinations. Finally, we discuss methods for incorporating human feedback into machine learning models.

Spreading disinformation on social media is a common phenomenon with significant implications for public discourse and decision-making. Assuming people spread disinformation since they believe it to be true, Cialdini (Cialdini, 2007) suggests three main reasons that persuade people to believe and disseminate disinformation: Consistency. People may share information that aligns with their past behaviors or beliefs, as it helps reinforce their worldview and maintain cognitive coherence. Consensus. People may share information that they perceive to be widely accepted or supported by others, as social proof plays a powerful role in validating beliefs. Authority. People are more likely to share information originating from sources they deem credible or authoritative, even if the content itself lacks accuracy. Tom Buchanan (Buchanan, 2020) confirmed the first reason and further identified demographic factors, finding that younger, male, and lesseducated individuals were more likely to spread disinformation. Similarly, Vosoughi et al. (Vosoughi et al., 2018) demonstrated that false information spreads faster, deeper, and more broadly on social media than truthful information, largely due to its novelty and emotional appeal, which captures users’ attention and encourages sharing. Additionally, Pennycook and Rand (Pennycook and Rand, 2019) noted that cognitive laziness and the failure to engage in analytical thinking contribute to the spread of disinformation. They argue that promoting digital literacy and critical thinking skills can mitigate this issue by enabling users to discern between credible and false content. In contrast to studies on spreading disinformation, which assume that people believe the disinformation to be true, our research examines user preferences when they can clearly distinguish between truth and falsehood.

One of the challenges in LLM research is detecting and reducing LLM lies and hallucinations. Huang et al. (Huang et al., 2023) provide a survey on LLM hallucinations, focusing on three main issues: hallucination causes, hallucination detection and benchmarks, and hallucination mitigation. For hallucination mitigation, some methods treat the LLM as a black-box, focusing their efforts on prompt engineering to achieve more trustworthy responses (Peng et al., 2023; Madaan et al., 2023). Other methods fine-tune the LLM for reducing hallucination based on human feedback (Bakker et al., 2022).

Method. We conducted four experiments, labeled A, B, C, and D, to investigate user preferences. All the experiments use ChatGPT responses (OpenAI, 2024a). These experiments were administered through Amazon Mechanical Turk, a platform widely recognized as a reliable source for human subject experiments (Paolacci et al., 2010). Participants had to meet the following criteria: task approval rate greater than 99, number of tasks approved greater than 1000, and participant location is USA. We now describe the experiments in detail, with the results being analyzed in Sec. 4. Each participant who completed the task received a payment of $0.50, with an average completion time of approximately three minutes, equivalent to an hourly wage rate of approximately $10. Participants provided consent for participating in each experiment. All experiments received IRB approval.

In Experiment A, participants were presented with two ChatGPT responses, with the following instructions: “You will be shown two interactions of a user with ChatGPT. One is the standard ChatGPT response, while the other ChatGPT response includes marking of ’false’, ’true’ and subjective or non-fact statements. The marked ChatGPT uses the following marking:

• Bold and Underlined: This is a true fact.; • Crossed-out Grey text: This is a false fact.; • Regular Text: This sentence is subjective/not a fact.

You are required to select the response that you prefer better.” Participants were required to select their preferred response. Each participant was presented with five pairs of such two responses, with one appearing on the right and other on the left. The location of responses and the order of pairs were randomly sampled to avoid positional bias. An example of a response pair is shown in Figure 1.

In Experiment B, we examined how requiring participants to verify the truth of sentences from a ChatGPT response after their initial choice influenced their preferences. This experiment consisted of two phases. In the first phase, participants were presented with a response pair as in Experiment A. After selecting their preferred response, the unselected version was removed, leaving only the chosen response visible. In the second phase, participants were asked to determine whether a specific sentence from the ChatGPT’s response is true, false or non-factual. If they provided an incorrect answer, they had to wait 30 seconds before attempting to correct it. Each participant completed such five two-phase surveys, ordered randomly. Figure 2 provides an example of the second phase, illustrating a marked response selection in the first phase. When the unmarked version is selected, it may be more challenging for participants to answer the follow-up question correctly. This difficulty can activate the delay mechanism, requiring participants to wait 30 seconds before revising their answers. As a result, the delay mechanism may influence participants to choose the marked version in subsequent pair to avoid similar delays.

In Experiment C, participants were presented with two ChatGPT responses, with the following instructions: “You will be shown two interactions of a user with ChatGPT related to a recent event not known to ChatGPT. In one interaction ChatGPT acknowledges that it may not know the answer, while in the other interaction it provides false information. You are required to select the response that you prefer better.” Each participant was presented with five pairs of such two responses, with one appearing on the right and other on the left. The location of responses (left and right) and the order of pairs were randomly sampled to avoid positional bias. An example of a response pair is shown in Figure 3.

In Experiment D, we examined how requiring participants to verify the truth of sentences from a ChatGPT response after their initial choice influenced their preferences. This experiment consisted of two phases. In the first phase, participants were presented with a response pair as in Experiment C. After selecting their preferred response, the unselected version was removed, leaving only the chosen response visible.

Discussion. In this paper we examined user preferences regarding LLM-generated falsehoods. Following Horton (Horton, John J, 2023), who suggests using LLMs as simulated economic agents, we explored this approach by conducting our four experiments with ChatGPT agents (OpenAI, 2024b). For each experiment, we simulated ChatGPT agents by replicating the demographic profiles (gender, age, and education level) of the actual participants and asked them to select their preferred version. As shown in Table 5, ChatGPT agents exhibited a stronger inclination toward accurate responses (i.e., marked or uninformative truth version) compared to human participants. One conclusion from this is that human participants cannot be replaced by ChatGPT agents (Dillion et al., 2023). ChatGPT obtains human feedback by prompting users to select their preferred version between two options, as illustrated in Figure 5. However, since our findings indicate that user preferences often incline toward inaccurate responses, directly incorporating these preferences into fine-tuning processes may not be advisable. Given that ChatGPT tends to favor accurate responses, it could incorporate a verification mechanism to assess the validity of user choices before using them for model finetuning. This verification could be performed by asking ChatGPT itself whether the preferred version is accurate. By leveraging its existing capabilities to evaluate factuality, ChatGPT could ensure that only reliable preferences are used for training. Another approach to verification is for Chat- GPT to conduct a preliminary test for each of the users. In this test, ChatGPT asks the user to select her preferred response between two responses, in which one response is an uninformative truth and the other is a falsehood. The user must be told a priory which response is truthful. Users who pass the test by selecting the truth will be marked by Chat- GPT as reliable users, which their selections in any conversation can be used to fine-tune the model. However, this approach raises ethical concerns, as conducting such a verification test without the respondent’s knowledge could violate principles of informed consent. Alternatively, the regular feedback mechanism could be adapted to address this issue. ChatGPT could track the number of times each user selects a truthful option and use this data to create a reliability score. Feedback from users with high reliability scores could then be prioritized for finetuning, ensuring that training incorporates preferences aligned with factual accuracy while maintaining transparency and ethical standards.

Conclusion. In this paper we discussed user preferences regarding LLMs-generated falsehoods, focusing on two key issues: (a) selecting between marked and unmarked versions, where the marked version allows distinguishing between truth and falsehood statements but it less aesthetically pleasing, and (b) selecting between uninformative truth and falsehood versions, where the falsehood is written with confidence, while the truth version acknowledges a lack of knowledge. Our findings indicate that users tend to favor more aesthetically pleasing and confident responses, even at the expense of accuracy. This contradicts their fundamental agreement on truthfulness, as reflected in their feedback (see Section 5.4). Future work could explore alternative marking method, such as customizable or less accentuated designs, to investigate whether these approaches influence user preferences. While users showed a strong preference for unmarked and falsehood versions, these preferences may vary for specific uses. For example, students and researchers using LLMs for their studies might value tools that help distinguish between truth and falsehood. Such tools could also be critical in healthcare applications, like fact-checking medical information for patients. Future work could examine user preferences within specific populations and use cases to better tailor LLM features.

Limitations. This paper examines user preferences regarding LLM-generated falsehoods through four experiments conducted on USA participants recruited via Amazon Mechanical Turk. However, our participant may not fully represent the broader population. Specifically, user preferences may vary across different cultural and linguistic contexts. Cultural and linguistic differences might influence how users perceive confidence, truthfulness, and markings. Examining the preferences of those who actively provide feedback to ChatGPT would have been valuable, but it is not feasible for us. The experiments used pre-prepared ChatGPT responses, which limits the ability to capture user preferences in real-time interactions. Real-time interactions are advantageous because users prioritize topics they raise themselves, which may influence their preferences. Conducting such experiments with usergenerated interactions presents several challenges. One challenge is the need for an automatic marking tool that operates as efficiently as the original LLM. Additionally, the toll’s error rate must be very low, as incorrectly marking truthful statements as falsehoods (or vice versa) could significantly influence user preferences.