Do bigger AI models actually hold a conversation's thread better, or do they all lose focus on earlier turns the same way?

This explores whether bigger models hold on to earlier conversation turns better than smaller ones, or whether attention fades the same way regardless of size. The short answer is that this collection doesn't contain a study that measures per-turn attention decay across model sizes directly. What it does have is a set of findings about *why* attention drifts, and several of them suggest that size alone isn't the main factor.

The strongest clue is that some of attention's biases come from the architecture itself, not from scale. Transformer soft attention gives too much weight to content that is repeated or prominent in the context, whether or not it's relevant. That creates a feedback loop in which whatever gets said often in a conversation pulls more and more attention, even before RLHF shapes the model's behavior Does transformer attention architecture inherently favor repeated content?. A related finding: a handful of huge, input-agnostic activations act as hidden 'attention sinks' that pull probability onto specific tokens, and this shows up across model sizes and even in vision transformers Do hidden massive activations act as attention bias terms?. If the mechanisms that decide where attention goes look the same at every scale, you shouldn't expect a bigger model to escape them automatically.

The more interesting lever turns out to be architecture rather than parameter count. Scaling laws that factor in design choices like hidden size, the balance between MLP and attention layers, and grouped-query attention (GQA) predict real differences between models trained on the same budget Can architecture choices improve inference efficiency without sacrificing accuracy?. Sparse attention makes the point even more sharply. When it's trained in from the start instead of bolted on later, it matches dense attention at frontier scale while cutting compute dramatically Can sparse attention match dense models without retrofitting?, and at equal compute a larger sparse model beats a smaller dense one on long-context tasks Does sparse attention trade off quality for speed?. Titans goes further: it hands long-range recall to a separate memory module that prioritizes surprising tokens, so attention only has to handle the recent window Can neural memory modules scale language models beyond attention limits?. These all point the same way. How a model *remembers* matters at least as much as how big it is.

One finding you might not expect is that some of what looks like attention 'decaying' over turns is really the context filling up. In multi-turn research agents, unrestricted reasoning in one turn eats the context space that later turns need, so the agent gets worse at taking in new evidence. Capping reasoning per turn fixes it Does limiting reasoning per turn improve multi-turn search quality?. That's a budget problem, not a size problem, and a bigger model with the same context window would hit it too.

One caution: scale does change some behaviors in ways you wouldn't guess. For example, which output format RL post-training ends up locking in depends on model scale Does RL training collapse format diversity in pretrained models?. So it's possible that turn-by-turn drift varies with size in ways nobody has measured yet. The honest summary is that the collection points to architecture, attention biases, and context budgeting as the forces that matter, and leaves the size question itself open.