SYNTHESIS NOTE
Do fine-tuned language models actually learn optimization procedures?
Can RL fine-tuning teach LLMs to solve constraint-optimization problems through genuine reasoning, or does it merely sharpen pattern-matching? Testing on out-of-distribution variants reveals the mechanism.
The constraint-optimization study uses a clean diagnostic to separate procedure from pattern: an N-case test set (in-distribution power-grid topologies) and an N-1 test set (the same problems with one element removed, putting them out of distribution while keeping the structure recognizable). A model running the actual procedure should perform comparably on both. A model running pattern-match should perform worse on N-1.
Even under GRPO and constraint-satisfaction-reward training, models degrade markedly on N-1. The conclusion is that RL on outcome-based rewards does not install the missing procedure — it sharpens the template-matching strategy along the in-distribution axis. The model gets better at recognizing patterns it has seen and worse, relatively, at adapting to perturbed structure.
This is methodologically important because it provides a probe that other reasoning evaluations lack. Most benchmarks cannot distinguish "the model solved this" from "the model recognized this." The N / N-1 comparison forces the distinction by holding the problem class fixed while perturbing the instance. The drop is the memorization signature.
For practitioners, the diagnostic generalizes. Wherever a deployment cares whether a model is computing or recalling — clinical reasoning, legal-statute reasoning, scientific problem-solving — building an "N-1" counterpart of the canonical test set is a cheap way to surface memorization. The structure-shift probe is more informative than headline accuracy on the canonical set.
Inquiring lines that read this note 97
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can AI alignment serve diverse human preferences at scale? How can identical external performance mask different internal representations?- How do unstated constraints become invisible to training data distributions?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- Can universal function approximators be expensive to learn in practice?
- Can instance seeds work for tasks beyond language understanding benchmarks?
- Can partial solution traces convert unproductive hard samples into learnable training data?
- How does the pretraining distribution shape what LLMs find hard?
- Does correct model behavior guarantee internal alignment of learned objectives?
- How can safety-aligned parameters be protected during user-specific fine-tuning?
- What hidden costs emerge when you fine-tune models for a single domain?
- Why does fine-tuning change how models process retrieved context?
- How does behavioral fine-tuning differ from factual knowledge encoding in models?
- Why does fine-tuning fail to remove temporal contamination from pretraining?
- Does fine-tuning actually change model capabilities or only output distribution?
- What happens to base model capabilities when you apply finetuning?
- Can models adapt and combine search strategies beyond their training algorithm?
- How much does pretraining quality affect the modularity of fine-tuned models?
- Does preference tuning help or hurt the exploration of solution spaces in code?
- Does fine-tuning a small model match fine-tuning a large one?
- Which finetuning method works best across different task and data regimes?
- How do finetuning and pretraining improvements differ in their effects on model capabilities?
- Can prompt optimization alone inject knowledge models don't already have?
- Can prompt optimization inject genuinely new knowledge into a model?
- Can prompt optimization or fine-tuning inject knowledge models do not already contain?
- What knowledge can prompt optimization actually activate in trained models?
- Why does prompt optimization alone fail to inject genuinely new knowledge?
- How much alignment data does a language model actually need to specialize well?
- Do LLMs rely on surface heuristics instead of learning recursive grammar rules?
- What separates pattern matching from genuine language understanding?
- What breaks when you apply reinforcement learning after supervised fine-tuning?
- Can smaller models achieve domain expertise through focused RL training?
- Which recipe choices determine the asymptotic ceiling in RL training?
- How does LLM simulation of APIs avoid instability without sacrificing training signal?
- Does the pretrained model prior limit RL search capability more than the optimization algorithm itself?
- How do sparse parameter updates enable when-not-how training to work?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- Can fine-tuning ever teach semantic inference instead of amplifying training shortcuts?
- Does supervised fine-tuning improve accuracy while damaging the quality of reasoning?
- Does fine-tuning on NLI tasks amplify or reduce frequency bias in language models?
- Why does fine-tuning improve some capabilities while degrading others?
- Why does supervised fine-tuning degrade reasoning quality despite raising accuracy?
- How does task-oriented fine-tuning compare to preference tuning methods?
- How does data quality mismatch create reasoning degradation in supervised fine-tuning?
- Does supervised fine-tuning improve reasoning or just response formatting?
- What makes supervised fine-tuning worsen RL exploration later?
- How does preference learning differ from supervised finetuning for reasoning?
- Does fine-tuning improve domain accuracy at the cost of reasoning quality?
- Why does NLI fine-tuning amplify frequency bias instead of teaching inference?
- Can reasoning improvements be attributed when optimizer and scaffold are unknown?
- How does self-distillation differ from standard fine-tuning approaches?
- Can trained models encode programs more complex than their data-generating process?
- Does reasoning fine-tuning actually reduce a model's ability to abstain?
- Does reasoning fine-tuning actually harm a model's ability to abstain?
- Can reasoning fine-tuning improve both capability and instruction compliance together?
- Why does reasoning fine-tuning reduce a model's ability to abstain?
- Can you control LLM reasoning strategy without fine-tuning the model?
- Why do reasoning models fail to improve constrained optimization performance?
- Does fine-tuning push models toward reasoning shortcuts that bypass the chain entirely?
- Why do rare complex structures in training data harm LLM generalization?
- Why does genetic programming outperform direct LLM generation by 86 percent?
- What mechanism causes LLMs to plateau on numerical optimization tasks?
- What constraint satisfaction rate do LLMs achieve at scale?
- Can LLMs improve at simple deduction through different training approaches?
- Can LLMs simultaneously reason and optimize their own modules?
- Why do large language models outperform fine-tuned models once repeated items are removed?
- Can memory-based adaptation and gradient fine-tuning operate on complementary timescales?
- Does RL amplify existing reasoning or create genuinely new computational strategies?
- When does RL discover genuinely novel reasoning strategies versus timing optimization?
- Can approximate or noisy reference answers work for RL-based reasoning training?
- How do retrieval and fine-tuning trade off flexibility against training cost?
- When does training a memory model beat RAG or fine-tuning?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do fine-tuned language models actually learn opt…
Do large language models actually perform iterativ…
Do larger language models solve constrained optimi…
Does supervised fine-tuning actually improve reaso…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Do large language models actually perform iterative optimization?
Explores whether LLMs execute genuine numerical procedures like Newton-Raphson or instead pattern-match to memorized solution templates when solving constrained optimization problems.
same paper, the mechanism this diagnostic exposes
-
Do larger language models solve constrained optimization better?
Explores whether scaling LLMs—through more parameters, better training, or reasoning extensions—improves their ability to satisfy constraints in real optimization problems like power grids and portfolios.
same paper, the parent finding
-
Does supervised fine-tuning actually improve reasoning on optimization problems?
When SFT boosts benchmark scores on constraint-optimization tasks, does it genuinely improve the model's ability to find feasible solutions, or just its ability to format answers convincingly?
same paper, complementary memorization signature
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Train Long, Think Short: Curriculum Learning for Efficient Reasoning
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
- On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
- Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs
- Are Emergent Abilities in Large Language Models just In-Context Learning?
Original note title
N-1 out-of-distribution tests reveal that RL fine-tuned LLMs still rely on memorization for optimization problems