Can subtracting attention maps remove noise from context?

The standard Transformer over-allocates attention to irrelevant context. Visualized on a needle-in-a-haystack retrieval task, only a small fraction of the attention mass lands on the correct answer while the bulk is smeared across distractors. This is not a training defect so much as a structural property of softmax: because the weights must be non-negative and sum to one, every token gets some mass, and across a long context that residual noise floor accumulates into real distraction.

DIFF Transformer's fix is elegant because it is mechanical rather than learned. It computes attention as the difference between two separate softmax attention maps. Each map carries the same underlying noise; subtracting them cancels the common-mode noise while preserving the differential signal — exactly the common-mode-rejection principle behind differential amplifiers and noise-cancelling headphones. What survives is a sparse attention pattern focused on relevant tokens. Crucially it stays FlashAttention-compatible, so the fix costs no efficiency. This is a distinct solution to a problem the vault has already diagnosed: since Does transformer attention architecture inherently favor repeated content?, the over-attention that produces sycophancy and lost-in-the-middle behavior is architectural — and where that note treats it as a bias to work around, DIFF Transformer removes it at the mechanism. It converges with What mechanism enables models to retrieve from long context?: both point to sparse attention as the substrate of reliable retrieval, one by discovering the heads that already do it, the other by engineering the whole layer toward it.