Could superposed decoding algorithms maintain multi-task representation during generation?
This explores whether new decoding methods could keep an LLM's ability to hold several tasks at once alive through generation — rather than collapsing to a single task the moment it produces a token.
This explores whether new decoding methods could keep an LLM's ability to hold several tasks at once alive through generation — rather than collapsing to a single task the moment it produces a token. The corpus has a paper that names exactly this problem: LLMs genuinely do represent multiple complete, distinct in-context tasks simultaneously during inference, but autoregressive decoding forces them to commit to one after the first token, so the superposition never survives into the output Can LLMs handle multiple tasks at once during inference?. So the answer to your question is conditional: the multi-task representation already exists inside the model — the bottleneck is the decoding step, which is precisely the part you'd be redesigning.
The most interesting lead is that the collapse may be an artifact of left-to-right generation specifically. Diffusion-based LLMs use bidirectional attention and refine all positions at once instead of committing token-by-token; one paper shows reasoning and answer being refined along separate axes simultaneously rather than serialized Can reasoning and answers be generated separately in language models?. If the first-token collapse is caused by autoregression's forced commitment, a refinement-based decoder is the natural place to look for preserving parallel tasks — the architecture doesn't demand you pick a winner up front.
There's also evidence the model's internals are already more parallel than its outputs suggest. Transformers compute correct answers in early layers and then actively overwrite them to produce format-compliant tokens — meaning distinct computations coexist and one gets suppressed at the surface Do transformers hide reasoning before producing filler tokens?. And networks tend to implement separable functions in isolated subnetworks rather than tangling them together Do neural networks naturally learn modular compositional structure?. Both suggest the raw material for multi-task generation is structurally present; what's missing is a decoding rule that reads out more than one of these computations instead of forcing convergence.
A second family of approaches keeps tasks separate by composing them at inference rather than blending them into one set of weights. Self-adaptive models mix expert vectors dynamically at inference without interference Can models dynamically activate expert skills at inference time?, and decoding-time proxy tuning steers output distributions while leaving the base model's parallel knowledge intact Can decoding-time tuning preserve knowledge better than weight fine-tuning?. The Consensus Game reframes decoding itself as a negotiated equilibrium between two policies rather than a single greedy commitment Can generative and discriminative models reach agreement? — a hint that decoding can be designed as a multi-objective process instead of a one-task funnel.
The thing you might not have expected: the obstacle isn't representational capacity, it's the commitment rule baked into how we generate text. The model holds the superposition fine — autoregression is what flattens it. That reframes "could superposed decoding work?" from a question about model power into a question about decoder design, and points you toward bidirectional/refinement decoders and inference-time composition as the corpus's two most concrete bets.
Sources 7 notes
Large language models represent multiple complete, computationally distinct tasks simultaneously during inference—a macroscopic phenomenon separate from feature-level superposition. However, autoregressive decoding forces convergence to a single task after the first token, preventing practical multi-task generation.
ICE shows that bidirectional attention in diffusion LLMs enables in-place prompting—embedding reasoning directly in masked positions refined alongside answers. Answer confidence converges early while reasoning continues refining, allowing early-exit mechanisms to cut compute by 50% while maintaining accuracy.
Logit lens analysis shows models trained with hidden CoT tokens compute correct answers in layers 1-3, then actively suppress these representations in final layers to produce format-compliant filler output. The reasoning is fully recoverable from lower-ranked token predictions.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Transformer2 demonstrates that tuning only singular values within weight matrices produces composable expert vectors that dynamically mix at inference without interference, outperforming LoRA with fewer parameters and enabling continual specialization.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
The Consensus Game frames decoding as a signaling game where generator and discriminator must agree on answers. Equilibrium-Ranking finds their joint policy, enabling 7B models to match 540B model performance without fine-tuning.