Could one test catch AI mistakes anywhere they arise — the model, the workflow, the humans around it — or must each layer be checked separately?

This explores whether one measuring tool can catch AI errors wherever they arise (in the model, the workflow around it, the deployment, the humans and institutions) instead of needing a separate check at each level. The corpus's blunt answer is that nobody has built one yet. One note surveying the field finds that partial instruments exist only for individual conditions in isolated settings: chain-of-thought disclosure for whether errors are visible, incident counts for containment, rollback timing for recoverability. None of them bridges all the conditions or captures human and institutional factors (How can we measure whether AI errors stay visible and recoverable?). So the useful question is why stitching local checks together doesn't work.

The reason is that local checks measure the wrong property. In three separate systems, every step passes its own plausibility, alignment or protocol check while the end-to-end workflow still fails, because the local test verifies something different from what makes the whole safe (Can individual components pass safety checks if the system still fails?). Attackers exploit this gap. Skill scanners score each skill on its own, so an attacker can use scanner feedback to make every piece look innocent while the chain of skills still does the harm, reaching about 96% average attack success (Can attackers evade skill scanners by refining individual skills?). A tool that inspects one layer's parts can be passed by a system that is broken across layers.

Two ideas in the corpus point toward a single instrument. The first is to measure at the boundaries, not inside each layer. Infrastructure-side recording of authority-bearing transitions is a vantage point that sits beneath the individual components. It tells apart a task that merely exposes a hacking vector from a run that actually exercised it, so not every score from an exposed task has to be treated as suspect (Can runtime instrumentation distinguish hacking exposure from actual exploitation?). The second is to choose the unit of observation deliberately. One proposed design compares four scopes at equal review cost and false-alert workload: isolated actions, rolling windows, known groups, and episodes discovered after the fact. The note reports no results yet, so whether wider scopes improve protection is untested (Does added monitoring improve protection at acceptable cost?). Scope also changes what you see within a single layer. Step-level confidence catches reasoning breakdowns that a global average hides (Does step-level confidence outperform global averaging for trace filtering?).

Any cross-layer instrument also has to handle errors that travel. A model's own earlier mistakes sitting in its context raise its future error rate non-linearly, and larger models don't fix this (Do models fail worse when their own errors fill the context?). One layer's error becomes the next layer's input, so a useful reading has to follow errors over time and across handoffs, not just count them. Two further traps apply. A consistent reading isn't a reliable one: a fixed seed and zero temperature give the same output every time, but it is still a single draw from a distribution (Does setting temperature to zero actually make LLM outputs reliable?). And no universal ranking of which systems are most vulnerable comes from one quantity such as distance. Exposure depends on where the scoring defect sits and how well the system searches (Can distance alone rank which substrates resist reward hacking?).

Put together, a single instrument would need to sit at the transitions where actions take effect, watch over sequences instead of single steps, and report how many draws its readings rest on. The corpus doesn't show anyone doing all of that, and the human and institutional layer remains the least instrumented.