How can agents detect missing information before attempting to solve problems?
This explores whether AI agents can recognize that a problem is under-specified — that they're missing a key fact — *before* they charge ahead and solve it, rather than confidently answering the wrong question.
This explores whether agents can notice missing information before solving, not after. The most striking finding in the corpus is that these are two different skills that don't come bundled. A model can ace a fully-specified reasoning problem and still fall apart when one variable is quietly withheld — accuracy on identifying the right clarifying question to ask drops to 40-50%, because gathering information and executing a solution are separable cognitive operations Can models identify what information they actually need?. So 'good at problems' does not imply 'good at noticing what's absent from the problem.'
The encouraging news is that the missing-information radar can be trained. With reinforcement learning, models went from catching almost nothing — 0.15% — to flagging deliberately flawed math problems 73.98% of the time, proactively identifying gaps and requesting clarification instead of guessing Can models learn to ask clarifying questions instead of guessing?. But the capability is fragile: in untrained models, giving them more inference-time 'thinking' actually made them worse at spotting flaws, and only after RL did extra compute help. The radar has to be explicitly built, not assumed.
Why is it missing by default? Because agents are passive by design. Optimizing for the next-turn reward structurally strips out initiative — the model is rewarded for producing an answer now, not for pausing to say 'wait, I don't have enough to go on' Why do AI agents fail to take initiative?. Detecting a gap is itself an act of initiative, which is why the harder engineering question becomes *when* to interrupt without being a nuisance. Conversation analysis offers a surprisingly concrete framework here: 'insert-expansions,' the little clarifying sub-dialogues humans use mid-conversation, formalize the moments an agent should probe the user rather than silently chain tools toward a misread goal When should AI agents ask users instead of just searching?.
There's a deeper reframing worth noticing. Detecting missing information isn't only a front-door problem ('do I have enough to start?') — it's a continuous one that runs through the whole task. Checking intermediate reasoning states rather than just final answers raised task success from 32% to 87%, because most failures are process violations that surface mid-trace, not wrong final answers Where do reasoning agents actually fail during long traces?. In other words, 'what am I missing?' is a question worth asking at every step, not just step zero. And when the agent genuinely can't resolve a gap alone, the human-agent collaboration literature is blunt that there's no clean rule for *when* to defer — instead of solving the timing problem, systems distribute the judgment across multiple touchpoints like co-planning, action guards, and verification When should human-agent systems ask for human help?.
The thread that ties these together: gap-detection isn't a property of a bigger model, it's something you have to architect in — through training, through externalized verification and memory structures, through interaction designs that make asking cheap Where does agent reliability actually come from?. The interesting corner the corpus leaves you with is that an agent's biggest blind spot may be the things it was never shown: trained only on tidy expert demonstrations, it inherits the curator's imagination and never learns to recognize the messy, under-specified situations that fall outside it Can agents learn beyond what their training data shows?.
Sources 8 notes
Models achieving high accuracy on complete reasoning tasks drop to 40-50% accuracy identifying what clarifying question to ask when one variable is withheld. Information gathering and problem execution are separable cognitive operations.
Reinforcement learning training increased proactive critical thinking accuracy from 0.15% to 73.98% on deliberately flawed math problems. Notably, inference-time scaling degraded this ability in untrained models but improved it after RL training, suggesting the capability is learnable but fragile without explicit training.
Research shows next-turn reward optimization structurally removes initiative from models, but proactive behaviors like critical thinking and clarification-seeking are trainable (0.15% to 73.98% with RL). The core challenge is balancing proactivity with civility to avoid intrusion.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Reliability for long-trace reasoning comes from checking intermediate states and policy compliance during generation, not from scoring final outputs. Adding intermediate verification raised task success from 32% to 87% because most failures are process violations, not wrong answers.
Magentic-UI identifies co-planning, co-tasking, action guards, verification, memory, and multitasking as mechanisms that work around the lack of ground truth for optimal deferral timing. Rather than solving the timing problem directly, these mechanisms distribute decision-making across multiple touchpoints.
Research shows reliable LLM agents externalize three cognitive burdens—memory (state persistence), skills (procedural components), and protocols (structured interaction)—into a harness layer rather than relying on model scale alone. The harness unifies these externalities and eliminates the need for the model to solve the same problems repeatedly.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.