If an AI only shows you the cases it's unsure about, do you stay sharp enough to catch its mistakes?

This explores whether a person who only sees the cases an AI flags as uncertain keeps the skill and attention to catch real mistakes. The corpus has good evidence that routing helps decisions and a sharp warning about how oversight can hollow out, but it doesn't directly test whether routing wears down the human's ability to scrutinize. The two pieces of evidence pull in opposite directions.

The case for routing is strong on outcomes. In AutoResearchClaw, a confidence-routed mode where humans step in only at high-uncertainty points had an 87.5% accept rate. That beat full autonomy (25%) and step-by-step human review (50%), and the note credits selective intervention with avoiding the rubber-stamping fatigue of constant interruption Does targeted human oversight beat both full autonomy and exhaustive review?. Fewer, harder decisions plausibly keep a reviewer more alert. The flagging itself can be cheap: one approach samples several candidate actions and only escalates when they disagree When should an agent actually stop and deliberate?. The trigger doesn't have to be the model's own self-doubt either, since lightweight features of the question predicted when to retrieve about as well as uncertainty estimates did Can question features alone predict when to retrieve?. But an accept rate measures the quality of the results. It says nothing about whether the human is getting sharper or duller.

The case against is that capacity can vanish without anyone noticing. Oversight processes can stay in place on paper after an organization loses the expertise, time, access and standing needed for real review. Nominal oversight produces the same recorded approval as genuine oversight, so standard audits can't see the loss Can organizations lose scrutiny capacity while keeping oversight forms?. Applied to routing, a reviewer's sign-offs on flagged cases would look identical whether or not they can still catch a subtle error. My own inference, not something the notes test, is that a reviewer who never sees the confident majority has no baseline for what normal looks like. They also can't notice when confidence is misplaced. That matters because the filter is imperfect. Model confidence does track robustness on average Does model confidence predict robustness to prompt changes?, but calibrated abstention is undertrained in standard LLMs Can models learn to abstain when uncertain about predictions?. Confident-but-wrong cases would then bypass the human entirely.

The most useful lead is to treat scrutiny capacity as something you measure rather than assume. One note describes four mechanical safeguards for LLM judges: measuring correctness against human labels and using planted cases as alarms Can deterministic checks protect LLM judges from failure?. It was written for judges, not human reviewers, but the idea seems to carry over. You could slip known-answer cases into the flagged stream, occasionally sample the confident ones, and check whether the human still catches the errors.

Routing to uncertain decisions solves the attention problem, since reviewers aren't drowned in trivial approvals. Whether it preserves the capacity to scrutinize is unproven here, and by the logic of nominal oversight, ordinary approval records wouldn't show if it didn't.