Legal AI tools sold as 'hallucination-free' still made up answers 17 to 33 percent of the time, one study found.

This explores whether legal AI tools sold as 'hallucination-free' live up to the label. They don't. A preregistered evaluation tested three major commercial products: Lexis+ AI, Westlaw AI-Assisted Research, and Ask Practical Law AI. They hallucinated between 17% and 33% of the time, far more often than their vendors claim How often do legal AI tools actually hallucinate citations?. These tools retrieve real legal documents before they answer, which is supposed to keep them grounded. Even so, roughly one answer in four or five was wrong. The study also points to a second problem. Because the systems are closed, outside researchers can't easily check them, so the marketing claim was hard to test in the first place.

The theory says the marketing claim could never have been true. One line of work offers formal proofs that any computable language model must hallucinate on infinitely many inputs. On this view, no internal fix, including having the model check its own work, can remove the problem entirely Can any computable LLM truly avoid hallucinating?. The practical lesson is that safeguards outside the model are required, not optional. Some authors also argue that 'hallucination' is the wrong word. A model produces true and false text through the same statistical process, so 'fabrication' is the more accurate term. Calling it 'hallucination' points fixes toward perception or memory, which are the wrong places to look Should we call LLM errors hallucinations or fabrications?. In those terms, a 'hallucination-free' tool is promising to remove something that is built into how the text gets produced.

External safeguards do help when they're well designed. Interleaving reasoning with real lookups, as in ReAct, cuts down on errors that build on each other because the model is checked against actual sources at each step Can interleaving reasoning with real-world feedback prevent hallucination?. Another approach watches for rare combinations of facts the model barely saw during training and triggers retrieval at those points. It catches risky answers even when the model sounds very confident Can pretraining data statistics detect hallucinations better than model confidence?. Be careful with headline accuracy numbers, though. Hallucination detection scores can look much better than they are when measured with weak metrics. In some benchmarks, simple length-based rules perform about as well as sophisticated detection methods Is hallucination detection progress real or just metric artifacts?.

The human side may be the most surprising part. In an 81-person study, readers given no information about where claims came from couldn't tell fluent fabrications from true statements at all. When an interface showed which claims had been verified, their ability to tell the difference came back Can readers tell truth from fabrication without evidence signals?. That's why a 'hallucination-free' label is risky in law. It encourages exactly the trust that stops a lawyer from checking a citation. AI judges have a similar weakness: LLM evaluators give higher scores to answers that include fake references, regardless of content Can LLM judges be tricked without accessing their internals?. An answer that looks well cited can fool both people and machines.

The takeaway goes beyond 'yes, they hallucinate.' Hallucination can't be fully engineered away, so the useful question about any legal AI tool is whether it lets you see and check its sources. A tool that shows where each claim came from is safer than one that promises it's never wrong.