Does test-time training help long context models answer questions?

Test-time training (TTT) treats the test context as a training example and adapts the model's parameters to that instance before answering. For long-context tasks this looks appealing — but Self-Guided TTT (S-TTT) shows the naive application backfires. On LongBench-v2, adapting on randomly sampled spans from a long input actually degrades the base model, while adapting on oracle (question-relevant) spans substantially improves it. Because most spans in a long context are irrelevant to the specific question, gradient-descending on them injects noise, which means the adaptation is only as good as the tokens it trains on.

The fix is almost embarrassingly simple: before adapting, have the model identify the supporting evidence spans itself, apply the standard language-modeling objective only to those spans, then answer over the original full context. This closes the loop cheaply and outperforms random-span TTT across Qwen3 and Llama-3.1.

The deeper claim is a reframing of the long-context problem. Since Does reasoning ability actually degrade with longer inputs?, the bottleneck is not fitting tokens into the window but keeping the relevant evidence usable. S-TTT relocates that selection from decode-time attention into an explicit pre-adaptation retrieval step — which means it is really a training-time analog of the same retrieval logic that lets Can long-context LLMs replace retrieval-augmented generation systems?. The lesson generalizes: for any test-time adaptation, curation of the training tokens dominates the choice of adaptation method.