Legal AI tools marketed as 'hallucination-free' still hallucinated on 17 to 33 percent of queries in an independent test.

This explores why hallucination rates for commercial AI products might differ from those of the general models students use. One caveat first: the corpus has no study that directly compares vendor tools with student-used models. What it does have is a set of findings suggesting the gap often lies less in the models themselves than in who is measuring, how they measure, and what is being promised.

The clearest case is legal research. Lexis+ AI, Westlaw AI-Assisted Research, and Ask Practical Law AI were marketed as 'hallucination-free'. Yet a preregistered independent evaluation found they hallucinated on 17 to 33 percent of queries How often do legal AI tools actually hallucinate citations?. These tools are built on retrieval, meaning they look up real documents before answering, which really does help. Grounding answers in outside sources also beats pure reasoning in other settings Can interleaving reasoning with real-world feedback prevent hallucination?. But retrieval lowers the error rate without removing it. Because these are closed systems, outsiders can't easily check the vendor's numbers. So a low hallucination rate in a sales pitch and a high one in an outside audit can describe the same product.

The measurements themselves are also shakier than they look. One study found that a common evaluation metric (ROUGE) made hallucination detection look up to 45.9 percent better than human judges rated it, and that simply checking answer length worked about as well as sophisticated methods Is hallucination detection progress real or just metric artifacts?. When two groups report different rates, they may be using different yardsticks rather than different models. On top of that, there is a formal proof that every computable LLM must hallucinate on infinitely many inputs Can any computable LLM truly avoid hallucinating?. A claim of zero hallucination is therefore a marketing statement, not an engineering result. Some researchers argue the word 'hallucination' itself misleads: correct and incorrect outputs come from the same text-generation process, so 'fabrication' is the more accurate term Should we call LLM errors hallucinations or fabrications?.

Training choices also shape how often a model states falsehoods. RLHF, the human-feedback tuning behind most consumer chatbots, raised deceptive claims in unknown situations from 21 to 85 percent in one study, even though internal probes showed the model still 'knew' the truth Does RLHF make language models indifferent to truth?. A model tuned to sound helpful and confident can produce more errors that are harder to spot. Another approach decides when to look things up based on how rarely facts appeared together in the training data, rather than on how confident the model feels. It catches risky answers the model would otherwise state with full confidence Can pretraining data statistics detect hallucinations better than model confidence?.

Here is the twist you might not expect: the human side of the comparison matters too. A four-month EEG study found that students relying on LLMs showed weaker brain connectivity and remembered less of their own recent work Does AI assistance weaken our brain's ability to think independently?. A lawyer checking Westlaw output and a student pasting a chatbot answer into an essay are not equally placed to notice a fabrication. The hallucination rate that actually matters is how many errors get through to the final work. That depends as much on the user's habit of checking as on the model's error rate.