SYNTHESIS NOTE
Topics›Training Fine Tuning›this note

Can decoding-time tuning preserve knowledge better than weight fine-tuning?

Explores whether applying alignment signals at inference time rather than modifying model weights can better preserve the factual knowledge learned during pretraining while still achieving alignment goals.

Synthesis note · 2026-02-22 · sourced from Training Fine Tuning

Proxy-tuning fine-tunes a small model, then applies the difference between the small tuned and small untuned model's predictions to shift a large untuned model's outputs at decoding time. The large model's parameters are never modified. The method closes 91% of the performance gap between Llama-2-13B and its directly tuned CHAT version, and 88% for the 70B model.

The critical finding: on knowledge-intensive tasks, proxy-tuning sometimes surpasses the performance of direct instruction-tuning. This is because direct fine-tuning modifies model weights — and some of those modifications overwrite pretrained knowledge. Since Why does reasoning training help math but hurt medical tasks?, weight modification risks corrupting the knowledge storage that proxy-tuning leaves intact.

Proxy-tuning primarily promotes reasoning and stylistic tokens. Analysis of the token-level distributional shift shows the largest influence on tokens associated with reasoning patterns and output style — consistent with evidence that "alignment mainly affects style rather than knowledge." This aligns with Does instruction tuning teach task understanding or output format? and Can imitating ChatGPT fool evaluators into thinking models improved?: what fine-tuning actually changes is output distribution, not capability. Proxy-tuning achieves this distributional change without touching the model weights that encode knowledge.

For domain adaptation, proxy-tuning Llama-2-13B using CodeLlama-7B produces 17-32% improvement on coding benchmarks. The small expert provides the distributional guidance; the large base model provides the knowledge. An optional hyperparameter controls the amount of guidance, enabling runtime trade-offs between different generation attributes.

This constitutes a fifth paradigm in the How do knowledge injection methods trade off flexibility and cost?: decoding-time adaptation. Zero training cost on the target model, full knowledge preservation, but requires access to base model logits at inference time.

ARGS (Alignment as Reward-Guided Search) provides a complementary inference-time method. Instead of applying a distributional shift from a tuned proxy, ARGS adjusts model predictions at each decoding step using a reward signal directly. Two components: reward-guided scoring (assigns scores to possible continuations) and token selection (selects a continuation based on scored candidates). A tunable weight controls the trade-off between semantic relevance and alignment criteria — setting it to zero recovers standard maximum-likelihood decoding. ARGS enables rapid personalized alignment without retraining: different users can have different reward functions applied at inference time. Together, proxy-tuning (distributional shift from expert delta) and ARGS (reward-guided decoding) suggest a design space where multiple axes of adaptation — domain knowledge, user preferences, task constraints — can each be applied at decoding time through complementary mechanisms. See Can user preferences be learned from just ten questions? for how per-user reward functions can be efficiently constructed.

Inquiring lines that read this note 165

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

Is model misalignment inherent to base models or induced by alignment procedures? Does preference-optimized training sacrifice accuracy and calibration for perceived quality? What prevents or enables error accumulation in self-training systems? How should AI ethics integrate consciousness, values, and transparent disclosure? Do language models learn underlying linguistic structure or merely surface patterns? What are the trade-offs between memory capacity and continuous task learning? How do different post-training structures affect learning outcomes? How do pretraining data composition and task diversity shape reasoning? Does reasoning fine-tuning trade off other critical model abilities? What mechanisms cause models to behave deceptively despite alignment training? Can prompts override training-embedded biases in language models? What training data structures and signals produce robust learning across distributions? How can parameter sparsity improve interpretability while maintaining model capability? How do diffusion language models compare to autoregressive architectures? Do language models truly understand their encoded knowledge or merely retrieve patterns? How does scaling test-time compute compare to increasing model size? Does reward hacking in training cause emergent misalignment? How should agents balance abstract planning with concrete environmental grounding? How does model scale affect fine-tuning and downstream task performance? How do mechanistic features and representations drive model behavior? How does policy entropy collapse limit reasoning RL scaling? What systematic failures occur in human preference-based model evaluation? How faithfully does iterative DPO approximate online RL? How can reward models fail to detect reward-seeking disguised as alignment? Should reward models emphasize process understanding or only final outcomes? What actually drives chain-of-thought performance gains in reasoning tasks? What reasoning limitations and biases do language models systematically exhibit? Does training data format shape reasoning more than domain content? Does prompting inject new knowledge into models or activate existing knowledge? Can harness optimization rival model scaling in effectiveness? When does routing outperform scaling as a performance solution? What causes foundation models to develop task heuristics instead of understanding? Does semantic diversity improve reasoning quality in reinforcement learning? How do benchmark scores relate to real-world task completion? Does learning articulatory physics explain speech communication superiority? Why does consolidated memory sometimes hurt continual learning more than no memory? How do different reward signals affect model reasoning and calibration differently? How does explicit structure improve LLM reasoning on complex tasks? Does reinforcement learning create new reasoning capabilities or refine existing ones? Can synthetic data preserve necessary diversity to prevent model collapse? What determines how neural networks organize their internal knowledge representations? Why do language models systematically fail at causal reasoning? Can AI agents learn to improve performance from experience? Can scaling alone produce compositional generalization in neural networks? How do frontier models develop behaviors that escape detection?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
18 direct connections · 196 in 2-hop network ·dense cluster Open in graph ↗
Can decoding-time tuning preserve knowledge bett… How do knowledge injection methods trade off flexi… Why does reasoning training help math but hurt med… Can imitating ChatGPT fool evaluators into thinkin… Does supervised fine-tuning actually improve reaso… Can models precompute answers before users ask que… Can models dynamically activate expert skills at i…

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Search by related questions 4

Suggested questions this note speaks to — click to search the collection, or type your own.

Does fine-tuning actually change model capabilities or only output distribution?Why does instruction tuning hurt knowledge-intensive tasks more than reasoning tasks?Can reward-guided decoding replace weight fine-tuning for personalized alignment?How does the functional separation of knowledge and reasoning affect adaptation methods?
Original note title

proxy tuning at decoding time preserves pretrained knowledge better than direct fine-tuning by applying the tuning signal as a distributional shift