Can gradient approximation at equilibrium replace backpropagation through time in practice?
This explores whether you can get useful learning signals from a settled equilibrium state — instead of unrolling and backpropagating gradients across every timestep (BPTT) — and the honest answer is that the corpus circles this question from the side rather than head-on.
This explores whether a model can learn or reason by relaxing to an equilibrium and reading gradients there, rather than backpropagating through a full unrolled sequence — the way recurrent nets are classically trained. None of these notes runs the literal head-to-head experiment (equilibrium propagation vs. BPTT), so if you came for that benchmark, the corpus doesn't have it. But it has a lot on the deeper instinct behind the question: that full backpropagation-through-time is expensive, brittle, and maybe avoidable.
The closest thing to your equilibrium framing is energy-based learning. Energy-Based Transformers assign an energy score to each input-prediction pair and then *descend* that energy surface at inference time — reasoning becomes iterative minimization toward a low-energy fixed point rather than a single forward pass Can energy minimization unlock reasoning without domain-specific training?. That's exactly the spirit of "compute the answer at equilibrium": the model settles into a solution by gradient descent on energy, and it reportedly scales better and generalizes further out-of-distribution than a standard Transformer. It's the corpus's strongest evidence that equilibrium-style dynamics can do real work.
The other lateral move is to ask: what if you skip parameter-space gradients altogether? A whole cluster of notes shows agents improving with *no* backprop at all. AgentFly treats learning as memory operations inside a memory-augmented MDP and hits strong benchmark scores without touching the model's weights Can agents learn continuously from experience without updating weights?, and Reflexion has agents write verbal self-diagnoses into episodic memory and improve across tries — credit assignment through language instead of through gradients Can agents learn from failure without updating their weights?. Titans pushes the same idea into architecture, splitting cheap short-term attention from a long-term neural memory that adaptively stores surprising tokens, sidestepping the quadratic cost that makes long-sequence backprop painful in the first place Can neural memory modules scale language models beyond attention limits?. So "replace BPTT in practice" has at least two live answers in this collection — settle to an energy minimum, or move learning out of the weights entirely.
There's also a quieter, surprising thread worth knowing: when you *do* use gradient-based RL, you're updating far less than you think. Across seven algorithms and ten model families, RL touches only 5–30% of parameters, and those sparse updates are nearly full-rank and nearly identical across random seeds — structural, not arbitrary Does reinforcement learning update only a small fraction of parameters?. That reframes the whole cost question: if the effective learning signal lives in a small, consistent subnetwork, the case for cheaper gradient approximations gets stronger, because most of the full backprop machinery may be doing redundant work.
The catch the corpus keeps flagging is plasticity. Methods that drift hard from the base distribution stall when the task changes, while staying close to base preserves the ability to keep learning Does staying close to the base model preserve learning ability?, and decoding-time proxy-tuning protects pretrained knowledge precisely *because* it never rewrites the weights Can decoding-time tuning preserve knowledge better than weight fine-tuning?. The takeaway you didn't come looking for: the interesting question may not be "equilibrium gradients vs. BPTT" at all, but whether you need to compute weight gradients on the fly — across this corpus, the most practical alternatives to backprop-through-time aren't better approximations of it, they're ways to avoid it.
Sources 7 notes
Energy-Based Transformers assign energy values to input-prediction pairs and use gradient descent minimization for inference, yielding 35% higher training scaling rates and 29% more inference-compute gains than Transformer++, while generalizing better on out-of-distribution data without domain-specific scaffolding.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.