Why do fixed-size document chunks break complex procedural question answering?
This explores why slicing documents into uniform, fixed-length pieces sabotages how-to questions that depend on ordered, multi-step procedures — and what the corpus offers instead.
This explores why slicing documents into uniform, fixed-length pieces sabotages how-to questions — the kind where step three only makes sense after steps one and two, and where a prerequisite buried elsewhere changes everything. The short version: fixed-size chunking cuts blind. It splits on token counts, not on meaning, so it routinely severs a procedure mid-stream — stranding a step from its prerequisite, or a branch from the condition that triggers it. The most direct treatment in the corpus replaces chunks entirely with 'logic units' — four-part structures of prerequisite, header, body, and linker — where the linker explicitly carries the navigation between steps and branches that chunking throws away How do logic units preserve procedural coherence better than chunks?. The key insight there is a double failure: embeddings confuse semantic similarity with task relevance, *and* chunk boundaries destroy sequential dependency. Procedural QA needs both fixed.
What makes this more than a chunking-strategy debate is that the same brittleness shows up at the model level, under different names. Reasoning accuracy collapses as inputs get longer — dropping from 92% to 68% with just a few thousand tokens of padding, far below any context-window limit Does reasoning ability actually degrade with longer inputs?. So even if you stopped chunking and dumped the whole procedure into a long context, the model would still wobble on the multi-step chain. And the wobble isn't really about 'difficulty': failures track instance-novelty, not task complexity, because models fit patterns from instances they've seen rather than learning a general algorithm they can replay on a new procedure Do language models fail at reasoning due to complexity or novelty?.
The long-context-versus-retrieval angle sharpens the picture. Long-context models can match retrieval systems on *semantic* lookup with no special training — but they fall apart on *structured* queries that require joining and relating discrete pieces Can long-context LLMs replace retrieval-augmented generation systems?. A complex procedure is exactly that kind of structured object: not a blob to match against, but a graph of steps and conditions to traverse. Throwing more context at it doesn't bridge the gap, and neither does naive chunking, because both treat the procedure as a bag of passages rather than an ordered structure.
There's a deeper reframe worth noticing. When reasoning models 'collapse' on multi-step procedures, the bottleneck is often execution bandwidth, not reasoning ability — text-only models that *know* the algorithm still can't run it at scale, while tool-enabled ones sail past the supposed cliff Are reasoning model collapses really failures of reasoning?. Read against the chunking problem, this suggests procedural QA fails on two fronts at once: retrieval destroys the structure on the way in, and the model strains to execute the steps on the way out. Fixing one without the other leaves you stuck.
If you want a doorway out, follow the structure-preserving thread — logic units that keep prerequisites attached and steps linked How do logic units preserve procedural coherence better than chunks? — and pair it with the recognition that step-execution itself may need offloading rather than more tokens Are reasoning model collapses really failures of reasoning?. The unifying lesson the corpus keeps returning: a procedure is a shape, and any system — chunker or model — that flattens the shape will break on the questions that depend on it.
Sources 5 notes
THREAD replaces chunks with four-part logic units—prerequisite, header, body, linker—enabling dynamic multi-step retrieval for how-to questions. Linkers explicitly navigate between steps and branches, addressing both the semantic-vs-task-relevance gap in embeddings and the sequential dependency loss in chunk-based RAG.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
The LOFT benchmark shows LCLMs match RAG on semantic retrieval without explicit training, but cannot execute relational queries requiring joins across structured tables. Context length alone cannot bridge this gap.
Models confined to text-only generation cannot execute multi-step procedures at scale, even when they know the underlying algorithm. Tool-enabled models solve problems beyond the supposed reasoning cliff, suggesting the bottleneck is procedural execution bandwidth.