Why do AI safety checks catch wrong answers but miss when a required warning simply never gets said?

This explores why standard safety checks fail to catch AI systems that quietly skip required clinical steps, as opposed to making loud, visible mistakes. The short answer from the corpus: most safety metrics look for wrong things that are present. A dropped obligation is a right thing that is absent, and an absence leaves almost nothing behind to detect.

The clearest example comes from clinical voice agents. When patients interrupted mid-sentence, four LLMs failed to deliver required safety content in 88.6–100% of cases, and telling them to be careful barely helped Do patient interruptions break clinical voice agent safety?. Nothing in these conversations looks broken. The agent sounds fluent and responsive, and the patient's interruption gets handled smoothly. What's missing, like a medication warning or a follow-up instruction, isn't marked anywhere in the transcript. A grader scoring tone, accuracy, or harmful content would pass it. Compare the AMIE study, where an AI took histories from 100 real urgent-care patients with zero safety stops Can conversational AI safely take patient histories without supervision?. That's a real result, but "no supervisor had to step in" measures whether something alarming happened. It doesn't tell you whether everything that should have happened did.

The broader argument is that safety failures hide in our evaluation habits rather than in technical mystery Why do safety failures remain invisible to our evaluation methods?. Deployed failures tend to be plausible rather than shocking, spread across a conversation rather than located in one turn, and absorbed by workflows that assume someone else caught it. A dropped obligation fits all three. Multi-agent systems show the same pattern outside medicine: agents that skipped a required verification step still reached verdicts matching the ground truth, so outcome-only monitoring couldn't tell compliance from corner-cutting Can a correct outcome hide protocol violations in multi-agent systems?. In clinical terms, the patient may be fine this time, and the skipped step only matters on the day it would have caught something.

The less obvious lesson is that fixing this is more a measurement design problem than a model problem. One paper finds that tools exist for each piece (chain-of-thought disclosure for visibility, incident counts for containment, rollback timing for recovery), but nothing measures whether an error stays visible and fixable across the whole human-plus-AI system How can we measure whether AI errors stay visible and recoverable?. Another proposes comparing monitoring that looks at single actions, rolling windows, or whole episodes, which matters here because a dropped obligation often only shows up when you look at the whole episode. It reports no results yet Does added monitoring improve protection at acceptable cost?. Shneiderman's framework helps explain why better models alone won't close the gap: reliability, safety, and trust come from different sources (engineering, management culture, independent oversight), and improving one doesn't improve the others Can we separate reliability, safety, and trust into distinct mechanisms?.

There is a hopeful counterpoint. In 39,849 Nairobi clinic visits, an LLM safety net cut diagnostic errors by 16% and treatment errors by 13% Can AI safety nets reduce errors in live clinical practice?. It worked partly because it ran as an asynchronous check on the clinician's completed work, which is the kind of whole-episode review that can notice something missing. The takeaway for builders is to turn obligations into an explicit checklist and score whether each item happened, rather than only scoring whether anything went wrong.