Can dataset design systematically expand reasoning graph diameter?
This explores whether deliberately constructing training data — rather than just scaling it — can stretch how many reasoning hops a model reliably chains together, i.e. expand the reach ("diameter") of its reasoning.
This reads the question as: can you *design* training data to lengthen the reasoning chains a model can traverse, instead of hoping longer reach emerges from scale? The corpus says yes, but with a sharp caveat about what the gains actually are. The most direct evidence comes from knowledge-graph curricula: fine-tuning a 32B model on 24,000 reasoning tasks *derived from medical knowledge-graph paths* produced state-of-the-art results across 15 domains, with the authors arguing that structured compositional knowledge mattered more than raw scale Can knowledge graphs teach models deep domain expertise?. In other words, walking longer paths through a graph and turning them into training instances is a deliberate lever on reasoning reach.
But here's the thing the question doesn't anticipate — work on *why* reasoning breaks suggests dataset design isn't really expanding a capability, it's expanding *coverage*. Reasoning failures turn out to be driven by instance-level unfamiliarity, not task complexity: a model will follow a chain of almost any length if it was trained on similar instances, and stumble on short ones it hasn't seen Do language models fail at reasoning due to complexity or novelty?. That reframes "expanding diameter" as "populating more of the reasoning-instance space." The same lens shows up in chain-of-thought research: CoT degrades predictably the moment you shift task, length, or format away from training distribution, producing fluent-but-invalid reasoning rather than transferable logic Does chain-of-thought reasoning actually generalize beyond training data?. So dataset design can systematically extend reach *within the distribution you build* — but the diameter is a property of your data's coverage, not a generalizable skill the model now owns.
There's also a hard ceiling the architecture imposes regardless of data. Reasoning accuracy collapses with input length far below the context window — dropping from 92% to 68% with just 3,000 tokens of padding, and chain-of-thought prompting doesn't rescue it Does reasoning ability actually degrade with longer inputs?. Longer reasoning graphs mean longer working state, so part of what limits diameter lives in the model, not the dataset.
If you do want to design data that targets reach efficiently, the corpus points at *where* the leverage is. Only ~20% of tokens are high-entropy "forking points" that actually carry the reasoning decision — and training on those alone matches full-gradient performance Do high-entropy tokens drive reasoning model improvements?. Dataset design aimed at diameter would do well to concentrate on these branch points rather than padding chains uniformly. And there's a tantalizing emergent angle: agentic graph reasoning self-organizes into a critical state where ~12% of edges stay semantically surprising even after structural connection, which keeps fueling new discovery Why do reasoning systems keep discovering new connections? — suggesting reach can also grow from the *process* rather than being pre-baked into the dataset.
The deeper lateral move is to question the premise. Several notes suggest the smarter play isn't lengthening chains inside the model at all, but externalizing them. Structuring reasoning as iteratively built knowledge-graph triples let GPT-4o mini jump 29% on hard multi-step tasks Can structuring reasoning as knowledge graphs help smaller models solve complex tasks?; symbolic rules derived from graph topology give models an explicit navigational plan across long hops Can symbolic rules from knowledge graphs guide complex reasoning?; and hypergraph memory binds three-or-more-entity relations so multi-step constraints survive across retrieval steps instead of decaying Can hypergraphs capture multi-hop reasoning better than graphs?. The thing you didn't know you wanted to know: you may not need to expand the model's internal reasoning diameter if you can offload the long path onto an external structure the model just navigates.
Sources 9 notes
Fine-tuning a 32B model on 24,000 reasoning tasks derived from medical knowledge graph paths produces state-of-the-art performance across 15 medical domains, demonstrating that structured knowledge composition matters more than scale.
LRMs don't break at complexity thresholds but at instance-novelty boundaries. Models fit instance-based patterns rather than generalizable algorithms, so any reasoning chain succeeds if trained on similar instances, regardless of length.
DataAlchemy experiments show CoT fails systematically under distributional shifts in task, length, and format. Models produce fluent but logically inconsistent reasoning — imitating reasoning form without valid underlying logic.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Analysis shows iterative graph reasoning evolves toward a stable phase where semantic entropy persistently dominates structural entropy, with ~12% of edges remaining semantically surprising despite structural connection, fueling ongoing discovery.
Knowledge Graph of Thoughts (KGoT) achieves 29% improvement on GAIA Level 3 tasks using GPT-4o mini by externalizing reasoning into iteratively constructed KG triples. The approach improves transparency, reduces bias, and enables quality control over reasoning steps.
SymAgent derives symbolic rules from KG structure using LLM reasoning to create navigational plans that align natural language with graph topology. This approach captures structural reasoning patterns explicitly, outperforming retrieval methods that rely on semantic similarity alone.
HGMem organizes retrieved evidence as hyperedges rather than flat lists or binary graphs, allowing three or more entities to bind into single relations without decomposition. This structure accumulates coherent knowledge across retrieval steps, trading representational complexity for constraint expressiveness.