Does secretly swapping in a human to play 'the AI' while testing live teach you more than grading the finished bot's answers later?

This explores whether testing an agent design live, with a hidden human standing in for the AI, teaches you more than grading a finished agent's outputs afterward. The short answer is that this collection has no papers on Wizard-of-Oz prototyping itself, so it can't make that comparison directly. What it does have is related evidence that feedback given during a live interaction shapes an agent differently from judgments made after the fact. That evidence points in a useful direction.

The closest match is Cocoa, a system that lets people revise an agent's plan while the agent is still carrying it out. Users can hand individual steps to themselves or to the agent as the work unfolds. In a lab study, people could steer the agent better this way than through an ordinary chat, and it didn't make the system harder to use Can users steer AI agents better by revising plans mid-execution?. This is the core idea behind Wizard-of-Oz testing: you learn what people need from an agent by watching where they want to step in mid-task, not by reviewing transcripts later. Cocoa builds those moments of stepping in into the product itself.

A second thread explains why the timing of feedback matters. Agents trained only on fixed sets of expert examples never learn from their own mistakes. Their skill is limited to whatever situations the people who built the dataset thought to include Can agents learn beyond what their training data shows?. Post-hoc evaluation has a similar blind spot: you can only score the situations you thought to test. A live session with a human in the loop surfaces situations nobody planned for. That is arguably the main thing a Wizard-of-Oz session gives you that a benchmark can't.

The collection also suggests where lessons from live sessions would go. A survey of self-improving agents splits improvement into two loops. One is a slow loop that retrains the model's weights. The other is a fast loop that edits prompts, memory and tools. Most recent progress has come from the fast loop because those changes are cheap and easy to undo Do self-improving agents really split into two distinct loops?. Related work finds that agents become reliable mostly by moving memory, skills and interaction rules into the surrounding software (the "harness"), not by using bigger models Where does agent reliability actually come from?. Improvements to that layer can lift a model's performance without retraining it at all Can execution harnesses lift model performance without retuning weights?. Things you notice in a live session, such as a missing clarifying question or a step users always want to take over, are exactly the kind of fix that belongs in the harness. That makes live testing well suited to how agents actually get improved today.

One caution: the system that edits the harness and the model that has to use it don't always match well. Models of every tier write useful harness changes about equally well. But mid-tier models benefit most from them, because weak models fail to use the changes and strong models don't follow them faithfully Do stronger models always evolve harnesses better?. So a fix that works well when a human plays the agent may not carry over cleanly once a real model takes the human's place. For a head-to-head test of Wizard-of-Oz methods against post-hoc evaluation, you would need to look outside this collection.