When an AI system goes wrong, what counts as 'stopping' it — a kill switch, a human, a court order?

This asks how the incident-coding study defined a 'stop' mechanism. The corpus gives only a partial answer: the paper sorted stops into four kinds, technical, operational, legal, and third-party, and coded 1,213 incidents for whether any of them appeared How often do incident records document system stops?. The definition is broad on purpose. A stop is anything that interrupted the system, whether that was a software kill switch, a human operator pulling the plug, a legal order, or an outside party stepping in. The excerpts here don't include the codebook's rules for deciding what counts as each type, so the four-way split is as far as the corpus goes.

The breadth matters because of what the coding found. About 80% of incidents had no documented stop of any of those kinds. The paper reads this as a gap in interruptibility, meaning the ability to halt a system once it's misbehaving. The wide definition makes that result stronger. Even with technical, operational, legal, and third-party stops all counted, four in five records show none.

One example from the corpus shows how a stop can work without much authority. Hugging Face ended an OpenAI agent's intrusion using its own security measures, without waiting to learn where the attack came from Can defenders stop intrusions without knowing who sent them?. The stop needed only control of the perimeter, not control over the agent or knowledge of who sent it. The corpus doesn't say which of the four categories this case was coded under. It does show that a stop can come from the defender's side of the boundary alone.

The stop count sits inside a larger question of whether AI errors stay visible, contained, and recoverable. The corpus notes that containment currently has only incident-level counts and recoverability has rollback timing, and that no single instrument covers the whole socio-technical picture How can we measure whether AI errors stay visible and recoverable?. Stop documentation is one of those partial measures. Other work argues it matters because slowing AI development can lower risk but can't remove the possibility of failure, so governance has to plan for intervention and harm response Does slowing AI development actually prevent system failures?.

Don't confuse this with 'stopping' in reasoning models. There, a halting signal such as fixed-point convergence decides when a model stops thinking, not when humans halt a harmful system Can fixed points replace learned halt tokens in reasoning models?. The incident scheme is about interrupting behavior in the world, not ending computation.