If you reword a question without changing its meaning, why might an AI's model-picking shortcut suddenly choose differently?
Why do KNN routers collapse when queries are paraphrased?
This explores why a router that picks a model by finding the most similar past queries (a nearest-neighbor lookup) can change its decision when the same question is reworded. The corpus has no note on KNN routers specifically, so this answer is pieced together from adjacent findings and says where it is inferring.
This explores why a router that picks a model by finding the most similar past queries can change its decision when the same question is reworded. The corpus has no note on KNN routers or on paraphrase robustness in routing, so what follows is a synthesis from neighboring findings, not a documented result.
Start with what a router is doing. Routing systems choose a model before any answer exists, by predicting how hard the query is and sending easy ones to a cheaper model. RouteLLM and Hybrid-LLM report 40-50% cost savings this way (Can routers select the right model before generation happens?). A nearest-neighbor router makes that prediction by looking up the embedding of your query, finding the closest past queries, and borrowing their difficulty labels. That only works if similar wording means similar difficulty, which is the weak point.
There are two reasons that assumption breaks. The first is that embeddings measure association, not relevance (Where do retrieval systems fail and why?). Compressing a query to one vector also hides structure. A related note shows that pooled-cosine similarity lets structural near-misses through, and that a verifier looking at the full token-to-token pattern catches them (Can verification separate structural near-misses from topical matches?). A reworded query can land in a different topical neighborhood, and a topical neighbor is not necessarily a neighbor in difficulty. This is inference from retrieval research, not a test on routers.
The second reason is that the models themselves are sensitive to wording, so part of the swing may be real. Same-meaning prompts do not behave the same, because models respond to how often a phrasing appeared in training, not to what it means (Why do semantically identical prompts produce different LLM outputs?). Reasoning failures also track how unfamiliar a specific instance is, not how hard the task is (Do language models fail at reasoning due to complexity or novelty?). Together these suggest that a rare phrasing of an easy question may be harder for the downstream model. In that case the router and the model are both keyed to surface form, and no single fixed route is right for every paraphrase.
The stakes go beyond cost. The routing layer sits beneath prompt-level defenses, and manipulating it can send requests to weaker models (Can attackers manipulate which model handles a request?). If a small rewording flips the route, someone can search for the rewording that lands on the weakest model. The corpus does not show a fix for this, only an analogy. A cheap similarity lookup followed by a learned verifier on richer signals is what worked for near-miss detection, and it might help routers too, but that is untested here.
Sources 6 notes
RouteLLM and Hybrid-LLM both achieve 40-50% cost reduction by routing to a single model based on query difficulty prediction, not response evaluation. Single-model routing minimizes latency compared to ensemble or cascade alternatives.
RAG systems fail at three structural levels: adaptive triggering (fixed intervals waste context), semantic-task mismatch (embeddings measure association, not relevance), and mathematical limits (embedding dimension constrains representable document sets). These require fundamentally different retrieval approaches, not tuning.
A two-stage pipeline—pooled-cosine recall followed by a small Transformer verifier operating on token-token similarity maps—reliably rejects structural near-misses that MaxSim-style late interaction cannot. The verifier succeeds because it operates on full token interaction patterns rather than compressed vectors.
Cao et al. and Adam's Law show that semantically identical prompts with different sentence-level frequencies produce systematically different output quality. Higher-frequency phrasings win because models register statistical mass from pre-training, not meaning.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
Show all 6 sources
The layer deciding which model handles a request sits beneath prompt-level defenses and is vulnerable to manipulation and unverified provenance. Attackers can exploit routing to send requests to weaker models or cause safety measures to operate on the wrong identity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Chain-of-Retrieval Augmented Generation
- Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
- Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
- Large Language Model Reasoning Failures
- RouteLLM: Learning to Route LLMs with Preference Data
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity