What human limits actually undermine AI oversight today?

The same framework paper on human oversight of AI argues that even where oversight is designed into a system, its effectiveness is undercut by well-documented limits of human-automation interaction, not just by poor interface design. Drawing on established human factors research (Kaber, 2025; Mosier and Manzey, 2019; Sheridan, 2021), it states that "human oversight is constrained by well-documented limits of human cognition, including reliance on heuristics (i.e., bounded rationality)" and that "oversight tasks also frequently involve prolonged periods of passive monitoring, a mode of engagement for which humans are poorly suited due to limitations in sustaining vigilance." It adds that explainability and transparency research has not closed the gap: these "do not guarantee meaningful understanding or reliable intervention."

The paper layers an organizational mechanism on top of the cognitive one: "organisational goals and incentive structures that undermine effective oversight, such as productivity targets that reward rapid acceptance of AI outputs rather than critical engagement" (citing Shankar et al., 2024; Parasuraman and Manzey, 2010). It also flags that the signals overseers must monitor are harder to read than in prior automation: AI monitoring "often involves interpreting probabilistic, multi-modal, and dynamically changing information signals rather than deterministic ones," and these signals "may drift over time, emerge only at aggregate levels, or be distributed across system components." The combined claim is that cognitive limits, incentive design, and signal opacity compound rather than operate independently.

This supplies a human-factors and organizational-incentive mechanism behind claims like Does granting agents more autonomy undermine human oversight? and Does AI augmentation protect workers from skill erosion?: both describe erosion over time, and this paper supplies a candidate cause independent of any particular agent's design — vigilance decrement and acceptance-rewarding incentives would degrade oversight even with a static, well-designed system. It also complicates Does labeling AI as an employee change how managers oversee it?: where that study locates the effect in framing and org-chart status, this paper's incentive-structure account suggests productivity targets could produce the same error-catching drop-off regardless of framing.

The excerpt names these as "persistent factors that impede human control" drawn from the cited human-automation literature; it does not report new measurements of how large the vigilance or incentive effects are for today's AI agents, nor does it test whether any specific interface or incentive redesign fixes them. The implication is that interface fixes alone — better explanations, more transparency — are not sufficient by the paper's own account, since it explicitly states these "do not guarantee meaningful understanding or reliable intervention": oversight design needs to also change what gets rewarded, not just what gets displayed.