Writing a paper's code, tables, charts and text is one thing; keeping them all telling the same story is much harder.
What makes consistency across code, tables, figures, and prose the hard part?
This explores why an AI (or a person working with one) struggles to keep a multi-part document, such as a paper with code, tables, figures and text, agreeing with itself, rather than just getting each part right on its own.
This explores why keeping code, tables, figures and prose in agreement is harder than getting any one of them right. The corpus has no note on that exact combination, so what follows is assembled from neighboring work on documents, dependencies and drift. The short version is that consistency isn't a property of any single piece. It lives in the links between pieces, and those links are where models fail quietly.
The first problem is that the failures are hard to see. In a study of long document-editing workflows, weaker models tended to delete content, which is easy to spot. Frontier models tended to corrupt it silently while keeping the surface looking intact (Does model capability change how documents degrade?). A table cell that no longer matches the sentence citing it, or a figure whose caption describes last week's version of the code, is that kind of failure. Everything reads fine locally, and the mismatch only shows up if you check across artifacts. Stronger models make that checking harder, not easier.
The second problem is that consistency is a web of dependencies, not a local property. Work on autoformalization makes this point for mathematics. Even a single theorem needs a coherent set of axioms, definitions and lemmas, and approaches that translate one statement at a time only look successful because they borrow the surrounding structure from a prebuilt library (Can autoformalization work on individual statements alone?). A paper has the same shape. A results claim depends on a table, which depends on a figure, which depends on the code that produced both. Handling each piece separately hides the actual work. The retrieval side shows a related failure. Fixed-size chunking cuts text apart without regard to what depends on what, and procedural coherence is lost. Explicit prerequisites and linkers that say how one step connects to the next help preserve it (How do logic units preserve procedural coherence better than chunks?). That work is about how-to steps, but the lesson carries over. If the connections between artifacts aren't represented, nothing holds them together.
A third factor is that a model isn't a single author holding one fixed version of the truth. One account describes an LLM as maintaining many possible characters at once, with each response sampling from that spread and narrowing as the conversation goes on (Does an LLM commit to a single character or maintain many?). Two passes over the same material can therefore land on different versions that are each plausible. When the code, the table and the paragraph are produced or revised at separate moments, they can drift apart. Each version fits the conversation so far, but they don't necessarily fit each other.
One more factor is that cross-references are structurally demanding. Models handle simple structure well and degrade predictably as nesting and dependency depth increase, which suggests they lean on surface heuristics rather than tracking the underlying structure (Does LLM grammatical performance decline with structural complexity?). That finding is about grammar, so applying it to documents is an inference on my part. Still, a claim in the prose that depends on a number in a table, which depends on a run of the code, is a deep dependency of the same kind. The corpus doesn't say whether a specific check, such as regenerating every artifact from one source of truth, fixes this, so that question is still open here.
Sources 5 notes
DELEGATE-52 shows weaker LLMs degrade documents through visible deletion, while frontier models degrade through subtle corruption that preserves surface integrity. This shift makes frontier failures harder to detect and potentially more dangerous at workflow scale.
Real formalization requires theory-level work: even one theorem needs a coherent web of axioms, definitions, and lemmas. Statement-level approaches only succeed by borrowing from prebuilt libraries like Mathlib, hiding the actual complexity involved.
THREAD replaces chunks with four-part logic units—prerequisite, header, body, linker—enabling dynamic multi-step retrieval for how-to questions. Linkers explicitly navigate between steps and branches, addressing both the semantic-vs-task-relevance gap in embeddings and the sequential dependency loss in chunk-based RAG.
Research shows LLMs don't commit to a single character but instead maintain a probability distribution over many consistent simulacra. Each response samples from this distribution, explaining why regenerations can yield different personalities while remaining consistent with prior context.
LLMs show systematic performance decline as syntactic depth and embedding increase. Simple sentences are handled well while complex structures with recursion and embedding fail consistently, suggesting LLMs learned surface heuristics rather than structural grammar rules.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Theory-Level Autoformalization: From Isolated Statements to Unified Formal Knowledge Bases
- Linguistic Blind Spots of Large Language Models
- Thread: A Logic-Based Data Organization Paradigm for How-To Question Answering with Retrieval Augmented Generation
- Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- LLMs Corrupt Your Documents When You Delegate
- Large Linguistic Models: Investigating LLMs' metalinguistic abilities