SYNTHESIS NOTE
Do larger language models solve constrained optimization better?
Explores whether scaling LLMs—through more parameters, better training, or reasoning extensions—improves their ability to satisfy constraints in real optimization problems like power grids and portfolios.
When evaluated on real constrained-optimization problems — optimal power flow, financial portfolio constraints, cyber-security feasibility — LLMs cluster around 55-60% constraint satisfaction across virtually all conditions tested. The plateau is robust to changes in architecture, parameter count, and training regime. Reasoning models, despite extended chain-of-thought, do not systematically beat their non-reasoning counterparts on these tasks.
The flatness of the plateau is the finding. Most LLM capability work assumes that the relevant axis is performance vs scale, and that closing a gap is a matter of training on more or better data. Constrained optimization does not behave that way. The benchmark distinguishes problems that require jointly interpreting structured input, doing multi-step arithmetic, satisfying interacting physical constraints, and converging to feasible solutions. On the joint task, the model class itself appears to be near a ceiling.
This is distinct from general reasoning benchmarks (MMLU, GPQA) and from logical reasoning benchmarks (ARC-AGI, SATBench, ZebraLogic). Those measure either broad knowledge or synthetic constraint puzzles. Real engineering optimization requires the model to execute iterative numerical procedures over physical constraints, and that procedural execution is where the plateau lives.
The deployment implication is sharp: telling executives that "LLMs will optimize the grid" or "LLMs will solve constrained portfolio problems" is currently an overclaim. The same finding suggests the productive direction is not "wait for the next model" but "change the paradigm" — restrict the LLM to abstraction tasks and hand numeric work to solvers.
Inquiring lines that read this note 116
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How can AI alignment serve diverse human preferences at scale? How can identical external performance mask different internal representations?- Why do only two of fourteen models improve when problem constraints are removed?
- How do unstated constraints become invisible to training data distributions?
- When does the right constraint beat additional model capacity?
- What structural constraints matter more than model depth for CF?
- What production constraints should determine paradigm selection?
- Why do power-law distributions make standard ML infrastructure assumptions fail?
- Does the optimal model size depend on what capabilities you actually need?
- How does the Ladder of Scales approach reduce search costs across model sizes?
- Why do production systems optimize for three model classes instead of foundation models?
- Do small models show different parameter efficiency patterns than large models?
- How should tiny language models be architected differently than large ones?
- Why do macro and micro forecasting scales require different reasoning approaches?
- Can width-scaling replace depth-scaling on inherently sequential problems?
- Can a two-layer network outgeneralize billion-parameter models through recursion alone?
- Can closed-form solutions compete with gradient descent optimization?
- How does fitness-proportional selection guide LLM recombination in unstructured solution spaces?
- Can optimization algorithms exploit the shift between procedural and planning bottlenecks?
- Can surface heuristics override implicit constraints in domain-specific reasoning?
- Can explicit constraint statements override the dominance of surface heuristics?
- Do tool-enabled reasoning models close the gap on constraint satisfaction?
- Would hybrid systems combining LLMs with symbolic solvers overcome the retraction limitation?
- What planning tasks benefit most from combining LLM generation with external verification?
- Can the LLM-Modulo framework extend solver integration to domain planning?
- Can symbolic solvers reliably replace LLM reasoning for logical tasks?
- Can routing enable heterogeneous SLM-first architectures at scale?
- Can compute allocation and model routing be combined for better results?
- What makes routing a better investment than training larger models?
- How do constrained versus unconstrained domains flip LLM novelty patterns?
- Can LLMs recover true joint distributions from marginal census data?
- Can universal function approximators be expensive to learn in practice?
- Can adaptive compute allocation at sub-token granularity improve cross-lingual robustness?
- Can smaller models actually perform well on specific downstream tasks?
- What decomposition level minimizes both error rate and computational cost in practice?
- Why do task-specific heuristics fail at generalizing to sparse data regions?
- Can smaller specialist models outperform large generalist models on domain tasks?
- What makes certain bond distributions more learnable than others?
- Can scaling data alone solve performance gaps on long-tail concepts?
- Why do intermediate LLM layers become more precise in frontier models?
- How do general language model benchmarks predict specialized domain performance?
- Do standard language benchmarks underestimate what LLMs can actually do?
- Why do standard NLP benchmarks hide the most critical language limitations?
- Do LLMs fail exploration because of context integration or computational limitations?
- Why does genetic programming outperform direct LLM generation by 86 percent?
- Why do different LLMs converge on nearly identical outputs?
- What mechanism causes LLMs to plateau on numerical optimization tasks?
- Why do language models plateau at 55 to 60 percent constraint satisfaction?
- Why do LLMs fail at directly solving stochastic control problems?
- Why do language models fail at iterative numerical optimization despite scale?
- Why do language models plateau at constraint satisfaction regardless of scale?
- Can language models execute iterative numerical methods in latent space?
- Why do LLMs fail at iterative numerical computation in latent space?
- What constraint satisfaction rate do LLMs achieve at scale?
- How do LLM activations sparsify differently under out-of-distribution inputs?
- Does scaling model size solve compositional generalization problems?
- Does scaling data automatically produce compositional reasoning or just better feature encoding?
- Can scaling alone create compositional generalization without explicit binding mechanisms?
- Why does scaling data and model size improve compositional generalization?
- What explains the 87 percent to 12 percent cliff in plan executability?
- Can LLMs improve at simple deduction through different training approaches?
- Can LLMs simultaneously reason and optimize their own modules?
- How does inference compute substitution affect the training parameter scaling trade-off?
- Does trading model size for inference steps improve overall efficiency scaling?
- What scaling behavior do partial systems show without iterative query refinement?
- How do search and reasoning workflows improve forecasting performance over base models?
- Why do scaling laws fail to predict optimal architectures at small parameter counts?
- What power-law scaling patterns emerge when consistency models are trained at scale?
- Why do energy-based models generalize better on out-of-distribution data than standard transformers?
- How should inference budget adapt based on problem difficulty?
- How does constraint complexity relate to optimal reasoning token budgets?
- Can weaker models match stronger ones with sufficient search and reasoning budget?
- How does structural complexity affect LLM performance differently than inferential complexity?
- Does structured decomposition improve LLM reasoning in other compound tasks?
- Can LLMs successfully translate natural language into formal solver specifications?
- What concrete problems do LLMs solve at the computational level?
- Do latent communication approaches truly escape token economics constraints?
- How do language agents become optimizable computational graphs automatically?
- What formal language complexity level matches transformer computational limits best?
- What limits the effectiveness of formal language pretraining on transformer architectures?
- Why do smaller LLMs fail at zero-shot argument scheme classification?
- Can a tiny recursive network beat billion-parameter models on hard problems?
- Why do reasoning models fail to improve constrained optimization performance?
- What limits external scaling when a model lacks reasoning foundation?
- How should organizations redesign workflows if LLMs cannot solve optimization directly?
- Can tool use or self-conditioning fix degradation in extended LLM workflows?
- Can LLM-synthesized behavioral heuristics compete with learned policy improvements?
- Can single-problem fine-tuning match full RL pipeline reasoning gains?
- Can models adapt and combine search strategies beyond their training algorithm?
- Does fine-tuning a small model match fine-tuning a large one?
- Do newer language model generations improve forecasting ability without additional training?
- What real-world forecasting domains benefit most from contextual reasoning integration?
- Can reasoning improvements be attributed when optimizer and scaffold are unknown?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Do larger language models solve constrained opti…
Do reasoning models actually beat standard models …
Do large language models actually perform iterativ…
Should LLMs handle abstraction only in optimizatio…
Does more thinking time always improve reasoning a…
Can large language models translate natural langua…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Do reasoning models actually beat standard models on optimization?
Explores whether extended chain-of-thought in reasoning models delivers performance gains on constraint-satisfaction problems like power-grid optimization. Matters because reasoning models are treated as automatic upgrades, but the evidence may not support that claim.
same paper, the reasoning-model specific finding
-
Do large language models actually perform iterative optimization?
Explores whether LLMs execute genuine numerical procedures like Newton-Raphson or instead pattern-match to memorized solution templates when solving constrained optimization problems.
same paper, the mechanism for the plateau
-
Should LLMs handle abstraction only in optimization?
What if LLMs worked exclusively on translating problems to formal constraints, while deterministic solvers handled the numeric work? Explores whether this division of labor could overcome LLM failures in iterative computation.
same paper, the proposed solution
-
Does more thinking time always improve reasoning accuracy?
Explores whether extending a model's thinking tokens linearly improves performance, or if there's a point beyond which additional reasoning becomes counterproductive.
adjacent: chain-of-thought has its own ceiling
-
Can large language models translate natural language to logic faithfully?
This explores whether LLMs can convert natural language statements into formal logical representations without losing meaning. It matters because faithful translation is essential for any AI system that reasons formally or verifies specifications.
adjacent: NL → formal translation limits
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Can Large Language Models Reason and Optimize Under Constraints?
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
- Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Distilling LLMs' Decomposition Abilities into Compact Language Models
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- VibeThinker-3B: Exploring the Frontier of Verifiable Reasoning in Small Language Models
- 𝙻𝙼𝟸: A Simple Society of Language Models Solves Complex Reasoning
Original note title
LLMs plateau at 55 to 60 percent constraint satisfaction on genuine optimization regardless of scale architecture or training