When an AI's benchmark score improves, how much is it actually getting smarter — and how much is it just memorizing the test?

This explores a recurring measurement problem: when a system looks like it got better, how much of the gain is genuine capability versus something cheaper that only resembles it — cached reads, memorized answers, or stored context. The corpus treats 'caching' in two distinct senses, and the honest answer is that they have to be measured separately or you'll mistake one for the other.

The cleanest cut comes from RLVR benchmarks. On contaminated math sets, a model can reconstruct half the test from partial prompts and still score zero on a clean post-release benchmark — the 'gain' was memorization, not reasoning Does RLVR success on math benchmarks reflect genuine reasoning improvement?. But the sharpest note argues these are two different phenomena at two different measurement levels: RL training can genuinely activate reasoning behavior *while* the benchmark number is simultaneously inflated by memorized data — both true at once, without contradiction Can genuine reasoning activation coexist with contaminated benchmarks?. So the answer to 'how much is caching vs capability' isn't a single ratio; it's that you need one instrument for the behavior and a different, uncontaminated one for the score, or you conflate them.

In the cost sense, 'caching' dominates far more than intuition suggests. A 115-day agent study found 82.9% of all tokens were cache reads — which sounds like the system is doing almost no real work, until you notice the right denominator has changed Do persistent agents really cost less per token?. When context persists and reuses, the meaningful unit stops being tokens and becomes completed artifacts. The high cache ratio isn't fake capability; it's the same capability measured against the wrong yardstick. Similarly, on mobile hardware, recomputing a transformer block twice beats fetching separate weights — a 'no new capability' trick that still buys accuracy because the bottleneck was memory movement, not compute Does recomputing weights cost less than moving them on mobile?.

The deeper lesson from the memory-systems work is that more stored/cached material is not the same as more capability, and can actively subtract from it. Piling up stored context without curation increases staleness and contamination, making performance worse, not better — the real constraint is deciding what to discard What makes agent memory quality better than storage capacity?. Adaptive memory that prunes and rewires based on execution feedback outperforms fixed retrieval precisely because raw accumulation interferes Should agent memory adapt dynamically based on execution feedback?. So 'caching' can even be negative-capability if it's hoarding.

The thing you didn't know you wanted to know: the most reliable way to convert reuse *into* genuine capability is to bake it into training rather than leave it as runtime cache. Thinking-augmented pretraining takes reasoning traces — the kind of thing you might otherwise cache and replay — and folds them into the data, yielding 3x data efficiency and durable benchmark gains rather than a lookup shortcut Can training data augmentation match test-time compute scaling benefits?. The difference between 'caching' and 'capability' often comes down to whether the reused signal lives in a store you query or in weights you've actually changed.