Most words an AI writes are filler — and that's not a trillion-parameter problem, it's how word-by-word generation works at any size.
Why does token redundancy and poor readability emerge at trillion-parameter scale?
This explores why large models produce so much filler and hard-to-parse text — but the corpus reframes the premise: redundancy isn't a side effect that appears at trillion-parameter scale so much as a structural fact about how token-by-token generation works at any scale.
This reads as a question about output quality at the frontier — why the biggest models still emit bloated, repetitive, hard-to-read text. The corpus doesn't treat this as a scale bug so much as a property of the medium: most tokens a model emits simply don't do much work, and that's true whether the model is small or enormous. When researchers prune reasoning chains by functional importance, a clear hierarchy falls out — symbolic computation tokens get preserved first, while grammar and meta-discourse ("let me think about this," "in other words") are the first to go Which tokens in reasoning chains actually matter most?. Redundancy, in other words, is the default texture of generated text, not an artifact of parameter count.
The sharper version of this comes from entropy analysis: only about 20% of tokens in a reasoning trace are high-entropy "forking points" where the model is actually making a decision. The other 80% are low-entropy continuations the model was always going to produce — and training on just the 20% matches or beats training on everything Do high-entropy tokens drive reasoning model improvements?. So the poor signal-to-noise ratio the question points at is measurable and, importantly, concentrated. The readable, load-bearing part of any generation is a minority of the tokens.
What's interesting is that some of the corpus suggests the redundancy isn't even at the model's level — it's at the tokenizer's. Byte-level models that segment text by next-byte entropy, spending more compute on surprising regions and coasting through predictable ones, match tokenized models while cutting inference cost Can byte-level models match tokenized performance with better efficiency?. The implication: fixed tokenization forces uniform compute across text that is wildly non-uniform in information content, which is a mechanical source of the waste the question describes. Redundancy is partly baked into the representation before the big model ever runs.
The corpus also complicates the idea that this is a simple trade-off you're stuck with. The apparent tension between generating diverse, information-dense output and generating correct output turns out to be a token-level measurement artifact — at the hidden-state level, exploration and exploitation barely correlate, and you can improve both at once Is the exploration-exploitation trade-off actually fundamental?. And redundancy across parallel generations can be attacked structurally: shared-prefix tree rollouts produce more genuinely distinct trajectories per token than independent sampling, because they stop re-paying for the same shared beginning Can shared-prefix trees reduce redundancy in agent rollouts?.
The thing you might not have expected: at persistent-agent scale, token redundancy stops being a quality problem and becomes an economic footnote. One 115-day study found 82.9% of tokens were cache reads — reused, not freshly generated — so the meaningful cost unit shifts from the token to the completed artifact Do persistent agents really cost less per token?. Redundancy at the token level is real, but the corpus keeps pointing to the same move: stop measuring at the token, and both the readability problem and the cost problem look different.
Sources 6 notes
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
The Byte Latent Transformer (BLT) dynamically segments bytes into patches based on next-byte entropy, allocating more compute to high-entropy regions and less to predictable ones. At 8B parameters, BLT matches tokenized baselines while reducing inference cost and improving robustness to typos and cross-lingual transfer.
Hidden-state analysis using Effective Rank metrics shows near-zero correlation between exploration and exploitation, revealing the trade-off emerges only at token level. VERL demonstrates simultaneous enhancement achieving 21.4% accuracy gains on Gaokao 2024.
Tree-structured rollouts that branch from shared prefixes produce more distinct trajectories within a fixed token budget than independent chain sampling. This improves advantage estimation statistics and enables longer-horizon tasks within the same compute constraint.
Show all 6 sources
A 115-day case study found 82.9% of tokens were cache reads. When context persists and reuses, the meaningful cost denominator becomes completed artifacts, not individual tokens.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
- From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
- How we built our multi-agent research system
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets
- Towards a Science of Scaling Agent Systems
- Do LLMs Encode Functional Importance of Reasoning Tokens?
- Byte Latent Transformer: Patches Scale Better Than Tokens
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning