SYNTHESIS NOTE
Do fine-tuned language models actually learn optimization procedures?
Can RL fine-tuning teach LLMs to solve constraint-optimization problems through genuine reasoning, or does it merely sharpen pattern-matching? Testing on out-of-distribution variants reveals the mechanism.
The constraint-optimization study uses a clean diagnostic to separate procedure from pattern: an N-case test set (in-distribution power-grid topologies) and an N-1 test set (the same problems with one element removed, putting them out of distribution while keeping the structure recognizable). A model running the actual procedure should perform comparably on both. A model running pattern-match should perform worse on N-1.
Even under GRPO and constraint-satisfaction-reward training, models degrade markedly on N-1. The conclusion is that RL on outcome-based rewards does not install the missing procedure — it sharpens the template-matching strategy along the in-distribution axis. The model gets better at recognizing patterns it has seen and worse, relatively, at adapting to perturbed structure.
This is methodologically important because it provides a probe that other reasoning evaluations lack. Most benchmarks cannot distinguish "the model solved this" from "the model recognized this." The N / N-1 comparison forces the distinction by holding the problem class fixed while perturbing the instance. The drop is the memorization signature.
For practitioners, the diagnostic generalizes. Wherever a deployment cares whether a model is computing or recalling — clinical reasoning, legal-statute reasoning, scientific problem-solving — building an "N-1" counterpart of the canonical test set is a cheap way to surface memorization. The structure-shift probe is more informative than headline accuracy on the canonical set.
Inquiring lines that read this note 109
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Is model misalignment inherent to base models or induced by alignment procedures?- Can communication problems and optimization problems be addressed with the same alignment approaches?
- Does correct model behavior guarantee internal alignment of learned objectives?
- How can safety-aligned parameters be protected during user-specific fine-tuning?
- What role does careful environment specification play in preventing misaligned optimization?
- How do unstated constraints become invisible to training data distributions?
- Can benchmarks designed for shortcut learning detect heuristic override failures?
- Can models adapt and combine search strategies beyond their training algorithm?
- How do different LLM integration paradigms affect inheritance of pretraining biases?
- How do LLMs compress specific expert knowledge into median abstraction?
- Why does genetic programming outperform direct LLM generation by 86 percent?
- Can you control LLM reasoning strategy without fine-tuning the model?
- What mechanism causes LLMs to plateau on numerical optimization tasks?
- What constraint satisfaction rate do LLMs achieve at scale?
- Can LLMs simultaneously reason and optimize their own modules?
- Does constraint-setting before generation change what LLMs can contribute?
- Can instruction tuning succeed without explicit task understanding?
- Can instance seeds work for tasks beyond language understanding benchmarks?
- Do LLMs rely on surface heuristics instead of learning recursive grammar rules?
- Do instruction-tuned models learn tasks or just output format distributions?
- What separates pattern matching from genuine language understanding?
- How should skill libraries coordinate with gradient-based weight optimization?
- Where does skill extraction fail compared to genuine model adaptation?
- What empirical evidence supports the Learning Law on real language models?
- Does instruction tuning optimize language models for rhetorical polish over logical consistency?
- What hidden costs emerge when you fine-tune models for a single domain?
- Why do production systems optimize for three model classes instead of foundation models?
- Why do fine-tuned models fail outside their specialized domains?
- Does fine-tuning actually change model capabilities or only output distribution?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- What filtering criteria best identify student-compatible refinements from teacher models?
- What happens to base model capabilities when you apply finetuning?
- How do retrieval and fine-tuning trade off flexibility against training cost?
- How much does pretraining quality affect the modularity of fine-tuned models?
- Does preference tuning help or hurt the exploration of solution spaces in code?
- Does fine-tuning a small model match fine-tuning a large one?
- Which finetuning method works best across different task and data regimes?
- How do finetuning and pretraining improvements differ in their effects on model capabilities?
- Can prompt optimization alone inject knowledge models don't already have?
- Can prompt optimization inject genuinely new knowledge into a model?
- Can prompt optimization or fine-tuning inject knowledge models do not already contain?
- What knowledge can prompt optimization actually activate in trained models?
- Why does prompt optimization alone fail to inject genuinely new knowledge?
- How does supervised finetuning change what prompting techniques actually accomplish?
- What breaks when you apply reinforcement learning after supervised fine-tuning?
- Can smaller models achieve domain expertise through focused RL training?
- How does LLM simulation of APIs avoid instability without sacrificing training signal?
- Does RL amplify existing reasoning or create genuinely new computational strategies?
- What makes supervised fine-tuning worsen RL exploration later?
- Does the pretrained model prior limit RL search capability more than the optimization algorithm itself?
- How do verifier-free RL patterns differ from traditional RLHF approaches?
- When does RL discover genuinely novel reasoning strategies versus timing optimization?
- How do sparse parameter updates enable when-not-how training to work?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- Can approximate or noisy reference answers work for RL-based reasoning training?
- Does RL primarily teach output format rather than new knowledge?
- How does fitness-proportional selection guide LLM recombination in unstructured solution spaces?
- Can LLM-based crossover and mutation work in unstructured natural language spaces?
- Can fine-tuning ever teach semantic inference instead of amplifying training shortcuts?
- Does supervised fine-tuning improve accuracy while damaging the quality of reasoning?
- Does reasoning fine-tuning actually reduce a model's ability to abstain?
- Does fine-tuning on NLI tasks amplify or reduce frequency bias in language models?
- Why does fine-tuning change how models process retrieved context?
- How does behavioral fine-tuning differ from factual knowledge encoding in models?
- Why does fine-tuning improve some capabilities while degrading others?
- Why does supervised fine-tuning degrade reasoning quality despite raising accuracy?
- How does data quality mismatch create reasoning degradation in supervised fine-tuning?
- Does reasoning fine-tuning actually harm a model's ability to abstain?
- Can reasoning fine-tuning improve both capability and instruction compliance together?
- Why does reasoning fine-tuning reduce a model's ability to abstain?
- Does supervised fine-tuning improve reasoning or just response formatting?
- Does fine-tuning push models toward reasoning shortcuts that bypass the chain entirely?
- How does preference learning differ from supervised finetuning for reasoning?
- Can reasoning improvements be attributed when optimizer and scaffold are unknown?
- Does fine-tuning improve domain accuracy at the cost of reasoning quality?
- Why does fine-tuning function as character training rather than capability training?
- Can LLMs improve at simple deduction through different training approaches?
- Does medical fine-tuning help LLMs through knowledge or reasoning ability?
- Why does fine-tuning fail to remove temporal contamination from pretraining?
- Why do large language models outperform fine-tuned models once repeated items are removed?
- How do out-of-distribution tests reveal that optimization learning is memorization?
- Can memory-based adaptation and gradient fine-tuning operate on complementary timescales?
- When does training a memory model beat RAG or fine-tuning?
- Which recipe choices determine the asymptotic ceiling in RL training?
- How does task-oriented fine-tuning compare to preference tuning methods?
- What concrete problems do LLMs solve at the computational level?
- How does the pretraining distribution shape what LLMs find hard?
- How do codebook length and data similarity affect accuracy in LLM coding?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do fine-tuned language models actually learn opt…
Do large language models actually perform iterativ…
Do larger language models solve constrained optimi…
Does supervised fine-tuning actually improve reaso…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Do large language models actually perform iterative optimization?
Explores whether LLMs execute genuine numerical procedures like Newton-Raphson or instead pattern-match to memorized solution templates when solving constrained optimization problems.
same paper, the mechanism this diagnostic exposes
-
Do larger language models solve constrained optimization better?
Explores whether scaling LLMs—through more parameters, better training, or reasoning extensions—improves their ability to satisfy constraints in real optimization problems like power grids and portfolios.
same paper, the parent finding
-
Does supervised fine-tuning actually improve reasoning on optimization problems?
When SFT boosts benchmark scores on constraint-optimization tasks, does it genuinely improve the model's ability to find feasible solutions, or just its ability to format answers convincingly?
same paper, complementary memorization signature
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Train Long, Think Short: Curriculum Learning for Efficient Reasoning
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
- Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs
- Are Emergent Abilities in Large Language Models just In-Context Learning?
Original note title
N-1 out-of-distribution tests reveal that RL fine-tuned LLMs still rely on memorization for optimization problems