Do users actually prefer truthful marked responses over confident falsehoods?

A study of 300 Amazon Mechanical Turk participants (US-based, task approval rate above 99, more than 1,000 tasks approved) tested whether users actually prefer truthful LLM responses when truth and falsehood are explicitly disclosed. In Experiment A, participants chose between two ChatGPT response formats: one with marking — "Bold and Underlined: This is a true fact," "Crossed-out Grey text: This is a false fact," "Regular Text: This sentence is subjective/not a fact" — and a standard unmarked response where truth and falsehood look identical. The paper reports "61% of users prefer unmarked falsehood responses over marked ones." In Experiment C, participants compared a ChatGPT response that "acknowledges that it may not know the answer" about a post-cutoff event against one giving a confident but inaccurate answer, and "69% prefer confident falsehoods over LLMs admitting lack of knowledge."

The authors attribute part of the unmarked preference to aesthetics: the marked version is truthful but "less aesthetically pleasing." A two-phase design (Experiments B and D) tested whether making users responsible for their choice changes it — after picking a response, participants had to judge whether a specific sentence in it was true or false, with a 30-second penalty for a wrong answer. Under this added accountability, the unmarked-preference result became "nearly evenly split" between versions, down from a clear one-phase majority — while the confident-falsehood preference from Experiments C/D remained high even with accountability. Because such real-time choices feed LLM development "through reinforcement learning from human feedback," the authors ask directly: "Does RLHF inadvertently encourage LLMs to generate inaccurate or false information?"

This sits alongside Do users worldwide trust confident AI outputs even when wrong?, but goes a step further: that cross-linguistic study finds overreliance arising from miscalibrated trust in confidence signals, while this study finds the same pull toward confidence over hedging even when the user has already been told which statement is false. It gives direct empirical weight to the concern in Are RLHF annotations actually measuring genuine human preferences?: here the elicitation artifact is not an ambiguous survey question but a disclosed-truth preference task, and users still chose the false, confident answer. It also complicates Do LLM explanations actually help people judge answer quality?, which found sources reduced trust in wrong answers — here, built-in truth marking only partially offsets the pull toward the unmarked, confident version.

The excerpt reports preference rates from pre-prepared ChatGPT responses rather than real, user-initiated conversations, and the sample is a US-only Mechanical Turk pool; the authors flag that preferences "may vary across different cultural and linguistic contexts." The findings establish a preference, not a demonstrated causal effect on any deployed RLHF pipeline — the RLHF risk is raised as a question for developers, not a traced mechanism. What the data do support: unless preference-based feedback is filtered or verified for accuracy before fine-tuning, feedback loops built on raw user choice risk rewarding confident falsehood over disclosed truth.