Why do 'human in the loop' systems give people a veto button but not the information to use it wisely?

This explores why human-in-the-loop systems tend to give people a way to step in (approve, reject, halt) without giving them a way to see and understand what the agent is actually doing. The clearest answer in the collection comes from Data & Society researchers, who call this the 'oversight fallacy' Does human-in-the-loop design actually enable effective oversight?. Builders treat intervention as if it were oversight. Approval checkpoints are concrete and easy to ship, and they're easy to point to as proof of control. Observation infrastructure (logs, traces, explanations of what happened) and knowledge infrastructure (the context a person needs to judge whether an action was right) are harder to build and less visible, so they get less investment. The result is a human with a veto and no basis for using it well. An approval button without understanding behind it is mostly a rubber stamp.

One useful way to see why this matters is to treat the human and the AI as a small organization, as Puranam does How should humans and AI systems divide decision-making tasks?. In that framing, splitting up the work is only half of the design problem. The other half is how each party learns, and that depends on what feedback reaches whom. A human placed at a checkpoint without good observation is a manager who signs off on reports they can't read. They can't spot patterns in the agent's mistakes, can't calibrate how far to trust it, and can't get better at overseeing it over time. Seen this way, neglecting observation isn't a missing feature. It's a broken learning loop in the organization.

A second, less obvious reason comes from the collection's critique of 'theory-free' AI Can AI models be truly free from human bias?. High accuracy can make a system look trustworthy enough that watching its reasoning seems unnecessary. That critique shows how a 95%-accurate system can still produce thousands of wrong outcomes, and those errors are only caught by someone who understands *how* the system reached its conclusions. If a design assumes that good aggregate performance plus an emergency brake is enough, it will underinvest in exactly the understanding needed to catch those failures.

To be direct about the limits: the collection has one note that addresses this question head-on, and the rest is adjacent framing rather than evidence about why builders make these choices. What you can take away is that 'can a human intervene?' and 'can a human oversee?' are different questions. The second needs observation and knowledge layers that most designs treat as optional.