Does attention linearity alone explain the efficiency gains over standard transformers?
This explores whether making attention linear (so cost grows in step with sequence length instead of with its square) is itself the source of efficiency gains — or whether the corpus shows efficiency coming from several other places too.
This explores whether "linear attention" is the single lever behind faster, leaner transformers, or just one move among many. The short answer from the corpus: linearity rarely travels alone. The systems that actually win on efficiency tend to pair it with other tricks, and several of the biggest gains come from places that have nothing to do with attention's math at all.
Start with the case that looks closest to "linearity is the answer." SpikingBrain converts an existing Qwen checkpoint into a near-linear-complexity model for long sequences — but notice it does so by combining linear/hybrid-linear attention *with* adaptive spiking neurons, and it leans on the fact that you can recover most of the performance with under 2% retraining (Can spiking neurons make transformers efficient on any hardware?). The efficiency story there is as much about cheap conversion and spiking sparsity as about the attention kernel. The honest counterweight is that pure linear-state models pay for their thrift: transformers provably beat state-space (fixed-size-state) models at copying and retrieving from context, because a compressed running state simply can't hold an arbitrarily long string the way full attention can (Can state-space models match transformers at copying and retrieval?). So linearity buys speed by throwing away the very capability that made attention expensive — it's a trade, not a free lunch.
That's why the more interesting architectures don't go fully linear; they split the job. Titans keeps quadratic attention for short-term, exact recall and bolts on a separate neural memory module that compresses and stores only "surprising" tokens, reaching 2M+ context without the quadratic penalty (Can neural memory modules scale language models beyond attention limits?). TransformerFAM gets unbounded-length processing by adding a feedback loop that lets the model attend to its own latents — and crucially adds *no extra weights* (Can models learn working memory by attending to their own latents?). In both, the efficiency comes from *what you choose to remember*, not from rewriting attention as a linear operation.
And some of the largest practical gains in the corpus sit entirely outside attention. MobileLLM finds that on memory-bound hardware the bottleneck is moving weights, not computing them — so sharing weights across blocks and recomputing them beats fetching fresh ones, a latency win with zero change to attention's form (Does recomputing weights cost less than moving them on mobile?). The Hierarchical Reasoning Model reaches problems fixed-depth transformers can't, with only 27M parameters, by using two recurrent timescales for effective depth (Can recurrent hierarchies achieve reasoning that transformers cannot?) — efficiency as architecture, not as attention arithmetic. Even within attention, there's a wrinkle that complicates any clean "linear vs. softmax" framing: a handful of input-agnostic "massive activations" quietly act as implicit attention bias terms (Do hidden massive activations act as attention bias terms?), and softmax attention carries structural baggage of its own, over-weighting repeated and prominent tokens regardless of relevance (Does transformer attention architecture inherently favor repeated content?).
So no — linearity alone doesn't explain the gains, and reading the corpus laterally, the better mental model is a portfolio: linear/hybrid kernels handle long-range cheapness, selective memory modules preserve recall, hardware-aware weight schemes cut data movement, and recurrent depth substitutes compute for parameters. The thing worth taking away is that "efficient transformer" is almost never one idea — it's a negotiated settlement between speed and the exact-recall ability that full attention uniquely provides.
Sources 8 notes
SpikingBrain successfully adapted Qwen2.5-7B using under 2% retraining data by combining linear/hybrid-linear attention with adaptive spiking neurons, achieving transformer-comparable performance with near-linear long-sequence complexity on non-NVIDIA hardware.
Two-layer transformers can copy exponentially long strings while state-space models are fundamentally limited by their fixed-size latent state. Empirically, transformers dramatically outperform SSMs at copying and context retrieval in both synthetic and pretrained settings.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
TransformerFAM demonstrates that adding a feedback loop lets transformers attend to their own latent representations, fostering emergent working memory for indefinitely long inputs. The approach requires no additional weights and improves long-context performance at 1B, 8B, and 24B scales.
MobileLLM shows that on memory-bound mobile hardware, sharing weights between adjacent transformer blocks by recomputing one block twice uses less latency than fetching separate weights, gaining accuracy with no parameter increase.
The Hierarchical Reasoning Model couples slow abstract planning with fast detailed computation across two timescales, achieving near-perfect performance on Sudoku and mazes where chain-of-thought methods fail completely. With only 27M parameters and 1,000 samples, HRM escapes the AC0/TC0 complexity ceiling that constrains fixed-depth transformers.
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.