SYNTHESIS NOTE
Do larger language models solve constrained optimization better?
Explores whether scaling LLMs—through more parameters, better training, or reasoning extensions—improves their ability to satisfy constraints in real optimization problems like power grids and portfolios.
When evaluated on real constrained-optimization problems — optimal power flow, financial portfolio constraints, cyber-security feasibility — LLMs cluster around 55-60% constraint satisfaction across virtually all conditions tested. The plateau is robust to changes in architecture, parameter count, and training regime. Reasoning models, despite extended chain-of-thought, do not systematically beat their non-reasoning counterparts on these tasks.
The flatness of the plateau is the finding. Most LLM capability work assumes that the relevant axis is performance vs scale, and that closing a gap is a matter of training on more or better data. Constrained optimization does not behave that way. The benchmark distinguishes problems that require jointly interpreting structured input, doing multi-step arithmetic, satisfying interacting physical constraints, and converging to feasible solutions. On the joint task, the model class itself appears to be near a ceiling.
This is distinct from general reasoning benchmarks (MMLU, GPQA) and from logical reasoning benchmarks (ARC-AGI, SATBench, ZebraLogic). Those measure either broad knowledge or synthetic constraint puzzles. Real engineering optimization requires the model to execute iterative numerical procedures over physical constraints, and that procedural execution is where the plateau lives.
The deployment implication is sharp: telling executives that "LLMs will optimize the grid" or "LLMs will solve constrained portfolio problems" is currently an overclaim. The same finding suggests the productive direction is not "wait for the next model" but "change the paradigm" — restrict the LLM to abstraction tasks and hand numeric work to solvers.
Inquiring lines that read this note 133
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Is model misalignment inherent to base models or induced by alignment procedures? How does model scale affect fine-tuning and downstream task performance?- Why do only two of fourteen models improve when problem constraints are removed?
- What production constraints should determine paradigm selection?
- Why do scaling laws fail to predict optimal architectures at small parameter counts?
- Why do power-law distributions make standard ML infrastructure assumptions fail?
- Does the optimal model size depend on what capabilities you actually need?
- Can smaller models actually perform well on specific downstream tasks?
- Can smaller specialist models outperform large generalist models on domain tasks?
- Why does adjusted compression performance degrade as models scale larger?
- How does the Ladder of Scales approach reduce search costs across model sizes?
- Why do production systems optimize for three model classes instead of foundation models?
- Do small models show different parameter efficiency patterns than large models?
- How should tiny language models be architected differently than large ones?
- Does fine-tuning a small model match fine-tuning a large one?
- Can externalized memory and skills replace model scaling?
- Why do small language models show faster on-premise payback than large ones?
- When does the right constraint beat additional model capacity?
- How does inference compute substitution affect the training parameter scaling trade-off?
- Does trading model size for inference steps improve overall efficiency scaling?
- Can weaker models match stronger ones with sufficient search and reasoning budget?
- Can closed-form solutions compete with gradient descent optimization?
- Can language models execute iterative numerical methods in latent space?
- What structural constraints matter more than model depth for CF?
- How do constrained versus unconstrained domains flip LLM novelty patterns?
- Would hybrid systems combining LLMs with symbolic solvers overcome the retraction limitation?
- How do LLMs compress specific expert knowledge into median abstraction?
- Do LLMs fail exploration because of context integration or computational limitations?
- How does structural complexity affect LLM performance differently than inferential complexity?
- Why does genetic programming outperform direct LLM generation by 86 percent?
- Can optimization algorithms exploit the shift between procedural and planning bottlenecks?
- What planning tasks benefit most from combining LLM generation with external verification?
- Can the LLM-Modulo framework extend solver integration to domain planning?
- Does structured decomposition improve LLM reasoning in other compound tasks?
- What mechanism causes LLMs to plateau on numerical optimization tasks?
- Can LLMs successfully translate natural language into formal solver specifications?
- How should organizations redesign workflows if LLMs cannot solve optimization directly?
- Why do language models plateau at 55 to 60 percent constraint satisfaction?
- Why do LLMs fail at directly solving stochastic control problems?
- What makes natural-language APIs particularly suited to LLM-based simulation?
- Can tool use or self-conditioning fix degradation in extended LLM workflows?
- Can symbolic solvers reliably replace LLM reasoning for logical tasks?
- Why do language models plateau at constraint satisfaction regardless of scale?
- Can smaller LLMs perform tool use tasks through modular decomposition?
- Can we systematically enumerate LLM failure modes from first principles?
- Why do LLMs fail at iterative numerical computation in latent space?
- What constraint satisfaction rate do LLMs achieve at scale?
- Can LLMs simultaneously reason and optimize their own modules?
- What capability boundary exists in LLM prediction of effect sizes?
- Does constraint-setting before generation change what LLMs can contribute?
- Can LLM-generated pages achieve quality parity with expert-designed interfaces at scale?
- Can surface heuristics override implicit constraints in domain-specific reasoning?
- How do unstated constraints become invisible to training data distributions?
- How do unstated feasibility constraints affect model decision-making?
- Can explicit constraint statements override the dominance of surface heuristics?
- Why do task-specific heuristics fail at generalizing to sparse data regions?
- Can models adapt and combine search strategies beyond their training algorithm?
- Can routing enable heterogeneous SLM-first architectures at scale?
- What makes routing a better investment than training larger models?
- Can universal function approximators be expensive to learn in practice?
- Can adaptive compute allocation at sub-token granularity improve cross-lingual robustness?
- How do LLM activations sparsify differently under out-of-distribution inputs?
- Can non-variational posterior approximation schemes deliver comparable reasoning improvements?
- How does nesting optimization levels improve on traditional network depth?
- Can width-scaling replace depth-scaling on inherently sequential problems?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- Can a tiny recursive network beat billion-parameter models on hard problems?
- Does scaling model size solve compositional generalization problems?
- Does scaling data automatically produce compositional reasoning or just better feature encoding?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- Why does scaling data and model size improve compositional generalization?
- Can scaling data alone solve performance gaps on long-tail concepts?
- What power-law scaling patterns emerge when consistency models are trained at scale?
- How do parameter scaling and latent vectors interact in language models?
- What explains the 87 percent to 12 percent cliff in plan executability?
- Can LLMs improve at simple deduction through different training approaches?
- Can measuring answer-space collapse show LLM narrowing in practice?
- Do tool-enabled reasoning models close the gap on constraint satisfaction?
- Why do smaller LLMs fail at zero-shot argument scheme classification?
- What limits external scaling when a model lacks reasoning foundation?
- Why do energy-based models generalize better on out-of-distribution data than standard transformers?
- What formal language complexity level matches transformer computational limits best?
- What limits the effectiveness of formal language pretraining on transformer architectures?
- How does fitness-proportional selection guide LLM recombination in unstructured solution spaces?
- Can LLM-based crossover and mutation work in unstructured natural language spaces?
- Can evolutionary search at inference time scale beyond natural language planning?
- What decomposition level minimizes both error rate and computational cost in practice?
- What planning strategies reduce execution steps without sacrificing solution quality?
- How should inference budget adapt based on problem difficulty?
- How does constraint complexity relate to optimal reasoning token budgets?
- Can compute allocation and model routing be combined for better results?
- Can compute budget scaling replace annotation budget in process supervision training?
- How do general language model benchmarks predict specialized domain performance?
- Can a model be strong at MMLU but weak at long-horizon tasks?
- Do smaller models optimized for benchmarks perform worse than their scores suggest in real tasks?
- Do standard language benchmarks underestimate what LLMs can actually do?
- Why do standard NLP benchmarks hide the most critical language limitations?
- What concrete problems do LLMs solve at the computational level?
- How do codebook length and data similarity affect accuracy in LLM coding?
- Why do different LLMs converge on nearly identical outputs?
- How much do different LLMs independently converge on similar outputs?
- Why does extended chain-of-thought reasoning fail to improve numerical optimization performance?
- Why do reasoning models fail to improve constrained optimization performance?
- Can citizen assemblies and value pluralism replace single utility optimization?
- Can small language models handle diverse tasks in heterogeneous multi-agent systems?
- Can LLM-synthesized behavioral heuristics compete with learned policy improvements?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- What class of RL problems can LLMs reliably turn into working reward code?
- Does RL scaling deliver much worse returns than inference scaling?
- Why do macro and micro forecasting scales require different reasoning approaches?
- What real-world forecasting domains benefit most from contextual reasoning integration?
- Do newer language model generations improve forecasting ability without additional training?
- How do search and reasoning workflows improve forecasting performance over base models?
- Can language models match competitive crowd forecasters on real future events?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do larger language models solve constrained opti…
Do reasoning models actually beat standard models …
Do large language models actually perform iterativ…
Should LLMs handle abstraction only in optimizatio…
Does more thinking time always improve reasoning a…
Can large language models translate natural langua…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Do reasoning models actually beat standard models on optimization?
Explores whether extended chain-of-thought in reasoning models delivers performance gains on constraint-satisfaction problems like power-grid optimization. Matters because reasoning models are treated as automatic upgrades, but the evidence may not support that claim.
same paper, the reasoning-model specific finding
-
Do large language models actually perform iterative optimization?
Explores whether LLMs execute genuine numerical procedures like Newton-Raphson or instead pattern-match to memorized solution templates when solving constrained optimization problems.
same paper, the mechanism for the plateau
-
Should LLMs handle abstraction only in optimization?
What if LLMs worked exclusively on translating problems to formal constraints, while deterministic solvers handled the numeric work? Explores whether this division of labor could overcome LLM failures in iterative computation.
same paper, the proposed solution
-
Does more thinking time always improve reasoning accuracy?
Explores whether extending a model's thinking tokens linearly improves performance, or if there's a point beyond which additional reasoning becomes counterproductive.
adjacent: chain-of-thought has its own ceiling
-
Can large language models translate natural language to logic faithfully?
This explores whether LLMs can convert natural language statements into formal logical representations without losing meaning. It matters because faithful translation is essential for any AI system that reasons formally or verifies specifications.
adjacent: NL → formal translation limits
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Can Large Language Models Reason and Optimize Under Constraints?
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
- Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Distilling LLMs' Decomposition Abilities into Compact Language Models
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models
- 𝙻𝙼𝟸: A Simple Society of Language Models Solves Complex Reasoning
Original note title
LLMs plateau at 55 to 60 percent constraint satisfaction on genuine optimization regardless of scale architecture or training