Who watches the automatic alarm when it fails?

Bainbridge draws on vigilance research to argue that an operator cannot be the monitor for low-probability events. Citing Mackworth (1950), she writes that "it is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour," so monitoring for unlikely abnormalities "therefore has to be done by an automatic alarm system connected to sound signals." The fix raises a new, unanswered question in the same breath: "who notices when the alarm system is not working properly?"

The regress is structural, not incidental. If the operator could reliably watch for rare abnormalities, no alarm would be needed; because the operator cannot, watching is delegated to automation — but that automation needs the same kind of watching the human was just excused from, so the monitoring problem reappears one level up rather than being solved. Bainbridge extends this with a related irony: automatic control can "camouflage" the very failure it should reveal, because it actively compensates for variable changes, so "trends do not become apparent until they are beyond control" — the system meant to make failure visible can instead be the thing hiding it. Her partial fix is a hybrid instrumentation design: "direct wired displays... for the main process information, and software displays for quantitative detail," so the primary signal does not depend on the same software whose failure it must reveal.

This gives a concrete, decades-old instance of the diagnosis in Why do safety failures remain invisible to our evaluation methods?: Bainbridge's camouflage point is a case where a failure is not "immediately legible" because the very system meant to surface it is actively masking it through compensation, not through any technical invisibility of the failure itself. It also sharpens Can organizations lose scrutiny capacity while keeping oversight forms? at the level of instrumentation rather than organizational capacity: even a fully resourced, attentive reviewer cannot be the backstop for the alarm layer, because sustained attention to a low-signal channel is not a capacity humans have, so "add a human reviewer" cannot close this particular gap no matter how much capacity that reviewer retains. Does automation actually remove humans from the control loop? is the companion irony from the same paper, about skill loss rather than monitoring regress.

Bainbridge's vigilance claim concerns visual monitoring of industrial process variables over roughly half-hour windows, backed by one 1950 study and her own design experience; it is not a claim about AI systems monitoring other AI systems, and the excerpt gives no figure for how reliable an automatic alarm layer itself needs to be, or how its own failures would be caught, beyond the wired-display suggestion. The implication the excerpt supports is narrow but still applicable beyond its era: whenever monitoring is handed to automation because a human cannot sustain it, the live question is not whether the handoff was correct but what, if anything, watches the automation doing the watching — a question the excerpt poses without answering.