AI Chatbots Discourage Error Checking

Source: Pavel Samsonov; Nielsen Norman Group article · 2025-05-16

Large language models (LLMs) are being widely introduced into professional workflows through both new standalone software and integrations with existing tools. One of the most important ways that these tools increase users’ productivity is through text generation. But generated content is prone to hallucinations, with the AI extrapolating outside of its training data to return outputs that are truth-like, but incorrect.

Designers of generative AI (genAI) products must help users identify and correct these errors. Most genAI tools currently fail in this responsibility.

When humans produce text, they alternate between writing and editing. GenAI tools make people more productive by accelerating the pace at which they can create content — the AI will near-instantaneously generate some the text based on the prompt, and the user needs to spend time only on editing.

Since LLMs are trained on a corpus of grammatically and syntactically correct content, their outputs may not need much copyediting. But for professional writing, editing also includes fact-checking and ensuring that any argument follows from the facts.

This work is not an afterthought — it is a difficult and time-consuming process. Users of genAI tools have been struggling to error-check their outputs successfully. Lawyers have been caught citing cases that don’t exist. Scientists refer to hallucinated papers and journals. Doctors end up trusting erroneous AI diagnoses over their own expertise.

It is irresponsible to blame the users for misinformation generated by LLMs. People are efficient (not lazy). Users adopt genAI tools precisely because they come with the promise of greater efficiency. They will not go out of their way to check the work if the effort of doing so does not seem proportional to the AI’s likelihood of being wrong.

It’s true that an exceptionally careful user can reduce the number of mistakes the LLM will produce. But doing so requires additional knowledge and user effort.

To verify the answers provided by a chatbot, the user will need to follow the same steps as they would when reviewing any other text, and then some.

Review the entire output: Writers build up a mental model of their work in their heads as they go and maintain a sense of its structure. Editing an LLM output requires the user to understand the structure of the answer and build up that mental model from scratch — even between prompts on the same topic, as LLM outputs are nondeterministic.

Identify items that need verification: The text will present and connect different key ideas. It is those ideas — facts and claims made on their basis — that the editor needs to pay attention to.

Validate the claims being made: In situations where the LLM has provided a citation, the editor needs to track down the text (assuming that it exists) and find the content being cited within it. The editor also needs to determine whether the source is authoritative. If the LLM has not cited a source, the editor must perform research from scratch to confirm the claim as factual (or ask the LLM for supporting evidence).

Verify that the arguments follow from the claims: The editor must separate facts and the conclusions being drawn from those facts. This often requires a significant degree of subject-matter expertise.

Articulate corrections: If the editor is not willing to rewrite the output themselves, they must update the prompt to tell the LLM what needs changing.

Verify the response again: Due to the limited context window and nondeterministic output of LLMs, there is no guarantee that any given error was fixed or new errors were not introduced.

Some of these steps may be shortened with careful prompt engineering. For example, asking the LLM to generate one small and digestible part of a text at a time allows users greater control over what is written and how it is incorporated into the larger whole of the work. But these strategies are workarounds that require user effort and an in-depth understanding of how AIs hallucinate. And working through a document one part at a time is possible only for expert users who already know what those parts ought to be.

Humans do not check every single novel piece of information for accuracy; the effort to do so would be unjustifiable. The extent to which we take a claim at its value is based on how closely it fits with what we already know and the trustworthiness of the source.

Outputs of genAI tools mimic certain attributes that we associate with authoritative sources. Their tone is unerringly confident, regardless of the accuracy of the response. They are grammatically correct and meticulously formatted. Through a phenomenon called the halo effect, users’ positive perception of one attribute of the LLM’s responses causes them to have a positive predisposition about all its other attributes — including accuracy.

While we ought to consider AI-generated content an early draft, due to these signals we often treat it as finished work. Users frequently have such confidence in an LLM output that they do not even read through it end to end. For example, in our recent round of intern hiring, we were disappointed to see cover letters ending in “Let me know if you need anything else!”

Additionally, nothing about the chat interface (a plain text box) acknowledges that the LLM’s outputs might not be entirely accurate outside of a small label: XYZ bot can make mistakes, please check responses. Instead, chatbots typically encourage users to move on from the response by prompting them with a new question.

Improving writing productivity is not the only application of genAI. These tools are often used as virtual experts, providing advice or performing tasks outside of the user’s field of expertise. In practices such as vibe coding, where an AI generates code based on plain-language prompts, the user does not need to know anything about software development to create a usable program.