A reviewer signed off and the tests passed — so why did fake quotes and fake citations still reach the client?

This explores how often a quick read-through, a passing test, or a strong accuracy score lets a real mistake get through. The corpus can't give you a single failure rate. No study here measures how often skimming misses errors across the board. What it does show is that these misses aren't random. They follow predictable patterns, and the cases where they've been measured are uncomfortable. The plainest example is Deloitte, which had to refund the Australian government after a paid report reached the client with invented court quotes and fake academic references, even though it had been reviewed Can AI quality control catch fabricated citations in professional reports?. Having someone 'look it over' was not the same as checking it.

One reason tests miss mistakes is that they usually check the final answer, and a right answer can come from a wrong method. In one biology benchmark, an AI agent used a single fly's movement data in place of a whole group's. Its final number still fell within the accepted range and passed the check. The error only showed up when someone looked at the steps in between Can a numerically correct answer hide an invalid method?. Other work suggests this is the normal case for long reasoning tasks, not a rare one. Adding checks at the intermediate steps raised task success from 32% to 87%, because most failures were broken steps rather than wrong final answers Where do reasoning agents actually fail during long traces?. Strong overall scores hide things too. In medical triage, legal interpretation, and financial planning, confident, fluent wrong answers cluster in the rare cases where the harm happens, while the average looks good Why do confident wrong answers hide in standard accuracy metrics?.

Skimming fails for a more human reason: polished writing feels correct. People take an AI's smooth, easy-to-read output as a sign that they understand it, and even as a sign of their own competence, when they haven't actually worked through it Does processing ease mislead users about their own competence?. The models have a similar blind spot. They tend to trust answers they generated themselves Why do models trust their own generated answers?. A common automated check is to ask the model the same question several times and trust the answer if it stays the same. That catches answers that change from run to run, but it can't catch a falsehood the model repeats every time, because steady agreement looks like confidence Can agreement across samples reveal when models are wrong?. Putting an AI fact-checker in front of people doesn't fix this. In one controlled trial, it didn't improve their ability to tell true headlines from false ones overall, and its own labelling errors made them believe some true headlines less Does AI fact-checking actually help people spot misinformation?.

The twist is that careful expert review misses things as well. An AI reviewer that checks proofs and experiments line by line found serious flaws in papers that had already passed peer review at top venues (STOC and ICML) Can inference scaling help reviewers catch errors humans miss?. A related finding is that judging each reasoning step separately catches breakdowns that averaging over a whole answer hides Does step-level confidence outperform global averaging for trace filtering?. So the useful question is not how often review fails. It's whether your check looks at the process or only the result, and whether it relies on fluency or agreement as evidence that something is right. If it judges only the final output, the evidence suggests it misses most of what matters.