When an AI breaks a task into steps to find the right tools, step count matters far more than word choice.
Why does fixing decomposition step count matter more than vocabulary alignment?
This explores why, when an LLM breaks a task into steps to fetch the right skills or sub-procedures, getting the *number* of steps right does more work than matching the exact words the retrieval system indexes on.
This explores why, when an LLM breaks a task into steps to fetch the right skills, getting the *number* of steps right beats matching the exact vocabulary the retrieval index uses. The corpus has a direct answer: standard LLM decomposition lands only around 34% step-level recall, and that low recall is the gate — nothing downstream can retrieve a skill for a step that was never carved out in the first place What blocks skill retrieval in task decomposition?. Correcting the step count recovers roughly 75% of the achievable gains, which tells you the failure was never that the model used the wrong synonym for a step. It was that the model lumped three operations into one, or split one into five, so the *units* it was searching for didn't correspond to the units the library stored.
Why does granularity dominate word-matching? Because vocabulary alignment is a reranking problem, and reranking only matters once the right candidate is in the pool. Fix the step boundaries and the bottleneck politely moves 'up' to representation-level reranking — you're now choosing among plausible matches rather than missing them entirely What blocks skill retrieval in task decomposition?. This is the same shape as the finding that a semantic-recall stage followed by a small learned verifier reliably rejects 'structural near-misses' that surface-similarity retrieval can't catch: the coarse pass gets you into the neighborhood, and the fine structural check is where the real discrimination happens Can verification separate structural near-misses from topical matches?. Structure is the load-bearing layer; wording is the cleanup.
There's a deeper reason the corpus keeps circling structure over surface: LLMs are strong at surface pattern-matching and systematically weak at the structural skeleton underneath. They misparse embedded clauses and complex nominals, and the errors get predictably worse as structural depth increases Why do large language models fail at complex linguistic tasks?, Does LLM grammatical performance decline with structural complexity?. The same brittleness shows up when models are asked to *execute* a procedure rather than name one: they recognize an optimization problem as template-similar and emit plausible-looking numbers instead of actually running the iterative steps Do large language models actually perform iterative optimization?. Decomposition step count is exactly where this weakness bites — carving a task into the correct number of executable stages is a structural judgment, not a lexical one, so it's the thing most likely to be wrong and the thing most worth fixing.
The lateral payoff: the corpus suggests you get more leverage from correcting *how a task is segmented* than from polishing *how each segment is phrased*. That pattern generalizes. Interestingly, one route to better decomposition may be to force the structure into the open — models that build explicit syntactic trees through step-by-step reasoning succeed where behavioral one-shot attempts fail Can language models actually analyze language structure?. If step count is the bottleneck, making the model reason about its own segmentation explicitly, rather than hoping better retrieval vocabulary papers over a bad split, is where the recall lives.
Sources 6 notes
Standard LLM decomposition reaches only 34% step-level recall, gating retrieval success. Correcting step count recovers 75% of gains in iterative methods, shifting the bottleneck to representation-level reranking rather than vocabulary alignment.
A two-stage pipeline—pooled-cosine recall followed by a small Transformer verifier operating on token-token similarity maps—reliably rejects structural near-misses that MaxSim-style late interaction cannot. The verifier succeeds because it operates on full token interaction patterns rather than compressed vectors.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
LLMs show systematic performance decline as syntactic depth and embedding increase. Simple sentences are handled well while complex structures with recursion and embedding fail consistently, suggesting LLMs learned surface heuristics rather than structural grammar rules.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Show all 6 sources
OpenAI's o1 model successfully constructs syntactic trees and phonological generalizations through explicit step-by-step reasoning, revealing that LLM linguistic capability extends far beyond behavioral language tasks to genuine language analysis.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Linguistic Blind Spots of Large Language Models
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Large Linguistic Models: Investigating LLMs' metalinguistic abilities
- Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways
- Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
- Can Large Language Models Reason and Optimize Under Constraints?
- Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose