Can any single continual learning method prevent forgetting?

The paper defines "long-horizon memorization": a model learns 100 query-answer tasks one after another through continual supervised fine-tuning, keeps no earlier training examples, and receives no task identifier at inference. Under that regime "sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon." The authors' response is a hypothesis rather than a fix already in hand: "mechanisms addressing complementary sources of forgetting will be more effective when composed."

The paper organizes the compositions along two design dimensions. Data, function, and weight anchors "specify what prior information each update should preserve," while low-rank allocation rules "determine where successive updates are retained." Crossing the two produces a combinatorial space, which the paper searches with task-level successive halving and then probes with a factorial experiment "to measure individual and interaction effects" across three distinct 100-task datasets. The logic is that forgetting has more than one source, so a mechanism aimed at one source leaves the others exposed. The interaction term in the factorial design is what would separate real composition from the sum of separate effects.

The setting is a deliberate bet on parameters as memory. Prompting and retrieval supply information at inference time, but it "remains outside the model's parameters and must be supplied again." That places the paper opposite Do memory systems actually help language models learn continuously?, where the failure is stale state in an external store. Here the failure is overwriting inside the weights. Both locate the problem in accumulation over many updates. Does staying close to the base model preserve learning ability? treats distance from the base model as one quantity to control, while this paper's design space suggests several separate things to preserve. Can splitting adaptation into two channels reduce forgetting? avoids the problem by moving part of the memory out of the weights, which this setting rules out by construction. Can models store unlimited facts without growing larger? adds a capacity argument against relying on weights alone. The excerpt does not engage with that bound.

The excerpt does not say how well any method retained anything. It gives no retention figures, effect sizes, model identities, or interaction estimates, so it cannot say whether composition beat the single mechanisms or by how much. The title asserts composition works, but the abstract frames it as the hypothesis under test. The discussion adds two limits. Evaluation uses "the queries seen during training," so the results "do not establish generalization to new query formulations." And "stronger memorization does not ensure preservation of general capability": the evaluated methods "still lose substantial accuracy on general capability benchmarks after 100 tasks."

Read at that strength, the paper is evidence that in-weight retention at a 100-task horizon is a multi-mechanism problem, and that what is retained is verbatim recall that still costs general capability. It is not evidence that weight updates can replace retrieval or prompting for information that must stay usable.