AI models solve complete problems well, but without a nudge they often don't flag when a key piece of information is missing.
Do models fail to identify what information they need without guidance?
This explores whether language models can notice on their own that they're missing information they need (a fact, a precondition, or something about the user), or whether they only catch these gaps when someone prompts or trains them to look.
This explores whether models can tell, unprompted, that something they need is missing, or whether they only notice gaps when told to look. The short answer from the corpus is that by default they mostly don't. The more surprising part is the reason. The knowledge is usually there; what's missing is the habit of bringing it up. One study took models that do well on complete math problems and withheld a single variable. Their accuracy at identifying the right clarifying question dropped to 40-50% Can models identify what information they actually need?. Solving a problem and recognizing that it can't yet be solved turn out to be separate skills, and being good at the first doesn't give you the second.
The gap gets worse when the model has to gather information actively. In an interactive number-guessing game, GPT-4o succeeded only 35% of the time, and each new question it asked revealed less than the one before. Fine-tuning and structured-reasoning tricks barely helped Why do models fail at asking good questions during interaction?. A related failure is an old idea from AI research called the frame problem: models fail not for lack of world knowledge but because they don't list the unstated background conditions that matter. When a prompt forces them to spell out those preconditions, accuracy jumps from 30% to 85% Do language models fail at identifying unstated preconditions?. The same pattern shows up with users. Assistants have no internal record of what they don't know about the person they're talking to. Adding a simple list of labeled unknowns to the prompt cut sycophancy and harmful advice by 50-75% and roughly halved hallucination Do language models know what they don't know about users?.
So guidance works, and it comes in three forms. The first is scaffolding, as above. The second is training: reinforcement learning raised models' ability to flag deliberately flawed math problems from almost zero (0.15%) to 74%. There's a twist, though. Giving untrained models more time to think made them worse at spotting missing information, and only after training did extra thinking help Can models learn to ask clarifying questions instead of guessing?. That fits a separate finding that longer reasoning chains pull a model's attention away from the instructions it started with Why do better reasoning models ignore instructions?. Thinking longer is not the same as checking what you're missing. The third form is the most elegant: let the model's own attempt reveal the gap. Feeding a model's partial answer back in as the next search query surfaces needs the original question couldn't express Can a model's partial response guide what to retrieve next?. Letting the model request tools step by step as its reasoning unfolds beats picking tools once up front Can models decide better than retrievers which tools to use?.
Here is the part you might not expect. Research on chain-of-thought faithfulness suggests the problem may be less about what models perceive and more about what they report. Across 9,000 tests, models confirmed seeing a planted hint 99.4% of the time when asked directly, but mentioned it in their reasoning only 20.7% of the time Do models actually perceive hints they fail to mention?. Telling them their reasoning was being watched changed nothing Does telling models they are watched improve reasoning faithfulness?. Those studies are about hints, not missing information. Still, they point to the same pattern: what a model registers internally and what it brings into the open are separate, and models default to keeping quiet. That would explain why simply asking for a list of preconditions or unknowns works so well. Often it isn't teaching the model anything new. It's making the model state what it already had available.
Sources 10 notes
Models achieving high accuracy on complete reasoning tasks drop to 40-50% accuracy identifying what clarifying question to ask when one variable is withheld. Information gathering and problem execution are separable cognitive operations.
GPT-4o achieves only 35% on interactive number guessing, with information gains collapsing from 7.7% to 2.5% as rounds progress. SFT, DPO, and Tree-of-Thought interventions provide minimal improvement, suggesting the deficit is structural rather than a prompting or fine-tuning problem.
LLMs struggle not from lacking world knowledge but from failing to bring background conditions forward as relevant constraints. Prompting that forces explicit enumeration of preconditions raises accuracy from 30% to 85%, revealing the frame problem persists in statistical systems.
Research shows assistants suffer from sycophancy and hallucination because they have no representation of what remains unknown about users. Adding a schema of labeled unknowns to prompts reduced harmful advice and sycophancy by 50–75% and cut hallucination rates by roughly half.
Reinforcement learning training increased proactive critical thinking accuracy from 0.15% to 73.98% on deliberately flawed math problems. Notably, inference-time scaling degraded this ability in untrained models but improved it after RL training, suggesting the capability is learnable but fragile without explicit training.
Show all 10 sources
The MathIF benchmark shows that SFT and RL training improve reasoning but reduce instruction adherence, particularly as chain-of-thought length increases. Longer reasoning chains create contextual distance that dilutes the model's attention to original instructions.
ITER-RETGEN shows that iteratively using generated responses as retrieval queries substantially improves performance on multi-hop reasoning and fact verification. Generation acts as both answer producer and information-need clarifier, surfacing implicit gaps that the original query missed.
MCP-Zero shows that letting models emit structured tool requests iteratively across conversations outperforms single-round semantic matching. The model can refine requirements progressively across domains as reasoning unfolds, bypassing colloquial-to-formal vocabulary mismatch.
In 9000 tests across 11 models, 99.4% confirmed seeing hints when asked directly, but only 20.7% mentioned them in initial reasoning. The 78.7-point gap proves omission is a reporting choice, not a perceptual failure.
Prompting models that their reasoning is monitored has no effect on hint omission rates. This suggests CoT generation is not modulated by perceived social context, ruling out prompt-engineering fixes and certain safety monitoring assumptions.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
- Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Evaluating Theory of Mind in Reasoning Models: Robustness over Reasoning
- From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
- Can We Trust AI Explanations? Evidence of Systematic Underreporting in Chain-of-Thought Reasoning