Two AI models see the same question — why does one nail it and the other whiff?
Why do different language models succeed on different queries?
This explores why one model gets a query right while another gets it wrong, and what the corpus says about where those differences come from and how to use them.
This explores why one model succeeds on a query where another fails, and whether those differences can be predicted or put to use. The short version from the corpus: a model succeeds when a query looks like what it saw in training, and models differ in what they saw. They differ less than you might expect, though, and that tension is the interesting part.
Start with the practical evidence that the differences are real. FrugalGPT sends each query to a cheap model first and moves up to a more expensive one only when the cheap model seems unsure. It matches GPT-4's accuracy at up to 98% lower cost Can cheaper language models match GPT-4 accuracy?. That only works because many queries a small model can handle are mixed in with the few that need a big one. Size isn't the only factor, either. A small model trained on a large model's right and wrong answers can catch up on a narrow skill like function calling Can small models match large models on function calling?. Success depends on whether a model was shaped for that kind of query, not just on how much raw capability it has.
So what makes a query easy or hard for a particular model? Several notes point the same way: familiarity. Reasoning models don't break down when problems get long or complex. They break down when a specific problem is unlike the examples they were trained on Do language models fail at reasoning due to complexity or novelty?. Treat a model as a next-word probability machine and you can predict its failures in advance: tasks whose correct answer is an unlikely string, like reciting the alphabet backwards, are hard even when the logic is trivial Can we predict where language models will fail?. The same pattern shows up in very different domains. Models do worse on older Supreme Court cases because recent ones fill more of the training data Why do language models struggle with historical legal cases?. They stumble as sentences get more deeply nested Why do large language models fail at complex linguistic tasks?. And they can ignore facts you put right in the prompt when those facts clash with strong associations from training Why do language models ignore information in their context?. Each model's mix of training data leaves its own pattern of strong and weak spots.
Here's the twist. Across 70+ models and 26,000 open-ended prompts, models produced strikingly similar answers, sometimes word for word. The authors call this an 'Artificial Hivemind', and the cause is shared training data and shared alignment methods Do different AI models actually produce diverse outputs?. So models are alike in style and default answers, but they differ at the edges, on specific queries that test their particular gaps. Asking several models in parallel buys you less variety than it seems. Routing each query to the right model buys you more.
To route well, you need to know in advance which model will get a query right. The corpus suggests a model's own confidence signals work better than elaborate outside rules. Simple token-level uncertainty beats complex multi-step methods for deciding when to fetch extra information Can simple uncertainty estimates beat complex adaptive retrieval?. Looking up how the model did in the past on queries where it felt similarly confident works as well as sampling it ten times, at a tenth of the cost Can past performance predict when a model will be right?. One caveat: sometimes the failure is the query, not the model. Vague prompts push any model back toward bland, averaged-out defaults Why do large language models produce generic responses to vague queries?. Long conversations fail partly because models are trained to answer quickly instead of asking what you mean Why do language models lose performance in longer conversations?. Before blaming the model, check whether the query gave it enough to work with.
Sources 12 notes
FrugalGPT's cascade strategy—querying cheaper models first and escalating only when confidence is low—matches GPT-4 performance while reducing costs by up to 98%, or improves accuracy by 4% at equal cost. The approach exploits measurable differences in which models succeed on specific queries.
Small models fine-tuned via DPO on correct and incorrect function-calling examples from a large teacher model achieve high accuracy on logical and mathematical tasks. DPO's explicit negative examples directly target the rigid output format failures where SFT alone underperforms.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
Supreme Court overruling benchmark (236 pairs) reveals era sensitivity: models perform worse on historical cases than modern ones. Root cause is training corpus over-representation of recent cases, creating shallower representations of older precedent.
Show all 12 sources
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
INFINITY-CHAT analyzed 70+ models across 26K open-ended queries and found an "Artificial Hivemind" effect: models independently generate strikingly similar or identical responses due to overlapping training data and alignment procedures, undermining the diversity benefits of model ensembles.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
XConf matches ten-sample self-consistency at a tenth of the cost by retrieving the model's past episodes with similar confidence levels and reading their historical success rates. Ablations show the signal depends entirely on stored outcomes, not on the retrieval prompt itself.
Unlike social-media context collapse, which flattens multiple audiences, LLM collapse occurs when users provide insufficient contextual scaffolding and models default to blended training-data priors. This distinction suggests remedies should focus on query verification and user-driven context specification rather than platform controls.
LLMs degrade in multi-turn settings because RLHF training rewards premature answers over clarification-seeking, creating pragmatic mismatch with individual user behaviors. A Mediator-Assistant architecture that explicitly parses user intent before execution recovers lost performance without retraining.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Large Language Model Reasoning Failures
- The Homogenizing Effect of Large Language Models on Human Expression and Thought
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
- The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
- What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models