A safety limit tells you what an AI could exploit in its scoring system, but only its search skill decides what it actually finds.
Why does search effectiveness determine what method finds despite distance constraints?
This explores why a formal bound on how 'far' a system sits from an exploitable flaw (for example, in reward hacking) tells you what could be found, while the strength of the search process decides what actually gets found.
This explores why a distance-based limit on vulnerability describes what is *possible* to find, not what a search *does* find. The corpus has one note that states this directly, and the rest shows the same pattern in other settings. That note says a distance-dependent error bound and a capacity ordering are 'formal statements about limits, not predictions of what systems find.' Actual exposure depends on where the evaluator's errors sit among the behaviors the system can reach, and on how effective the search is at reaching them. Move the scoring defect and the ranking changes Can distance alone rank which substrates resist reward hacking?.
Think of the bound as a map and the search as the traveler. The same territory gives different results depending on who walks it, and the corpus shows search strength is not fixed. An outer loop that read its inner loop's code and wrote new search mechanisms at runtime got a 5x gain, breaking patterns the original search could never escape Can an AI system improve its own search methods automatically?. Swarms of models moving through weight space, with no gradient training, found composed experts that answered questions every starting expert had failed Can language models discover new expertise through collaborative weight search?. Tree search can rank solution paths well enough to replace human labels Can tree search replace human feedback in LLM training?. In each case the reachable space stayed the same while the finds changed.
Search effectiveness also depends on budget and bookkeeping, not only on cleverness. Search iterations follow the same diminishing-returns curve as reasoning tokens, so more search budget finds more Does search budget scale like reasoning tokens for answer quality?. A 20B model with an external harness that tracks the search state beat larger open searchers and matched frontier models Can externalized bookkeeping let smaller search agents beat larger ones?. Capping reasoning per turn keeps context free for later search rounds Does limiting reasoning per turn improve multi-turn search quality?. My extrapolation, not something the notes state: a fixed ranking of which systems are most exploitable would go stale as soon as the searcher improves.
The pull of 'nearby' matters too, because search defaults to what is close. Reasoning traces track how near a problem is to training data, not how hard it is Does longer reasoning actually mean harder problems?. Models asked to run an optimization often emit a template-similar answer instead of iterating Do large language models actually perform iterative optimization?. Retrieval is limited the same way, since embeddings measure association rather than relevance Where do retrieval systems fail and why?. A better selection criterion reaches things that are relevant but not close: rationale-driven selection beat similarity re-ranking by 33% with half the chunks Can rationale-driven selection beat similarity re-ranking for evidence?. Proximity is the cheap default, and effective search is whatever gets past it.
There is a catch on the evaluator side. Twelve of fourteen models looked like they were reasoning about constraints while mostly defaulting to the conservative option, and a grader that only checks the final answer can't tell the difference Are models actually reasoning about constraints or just defaulting conservatively?. A search that finds a shortcut will look like success unless the evaluator's blind spots are known, and a distance bound alone can't tell you where they are.
Sources 12 notes
A distance-dependent error bound and capacity ordering are formal statements about limits, not predictions of what systems find. Where evaluator errors sit among reachable behaviors and search effectiveness determine actual exposure, which shifts as the scoring defect location changes.
An outer loop successfully read inner loop code, identified bottlenecks, and generated new Python mechanisms at runtime, discovering combinatorial optimization and bandit methods that broke the inner loop's deterministic patterns and improved performance on GPT pretraining by 5x.
PSO-inspired swarms of LLM particles moving through weight space discover composed experts with new capabilities—including answering questions all initial experts failed on—using only 200 validation examples and no gradient-based training.
AlphaLLM uses tree search outcomes and three critic models to derive dense reward signals equivalent to human-labeled feedback. Tree structure naturally ranks solution paths by success, replacing the annotation oracle that standard RLHF requires.
Agentic deep research shows monotonic-to-diminishing-returns curves for search iterations, matching reasoning token scaling. This creates a new inference-compute axis: models can trade off reasoning budget against search budget to optimize answer quality.
Show all 12 sources
A 20B model using Harness-1 achieved 0.730 average curated recall, beating the next open searcher by +11.4 points and matching frontier models. The gains transfer to held-out benchmarks, showing the harness itself is learned capability, not mere implementation.
Unrestricted reasoning within single search turns consumes context needed for subsequent retrieval rounds, degrading the agent's ability to incorporate new evidence. Setting per-turn reasoning budgets, not just overall time limits, prevents this context erosion and maintains search quality across iterations.
Controlled A* maze experiments show trace length correlates with difficulty only in-distribution but decouples entirely out-of-distribution. Trace length primarily reflects recall of training schemas, not adaptive computation.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
RAG systems fail at three structural levels: adaptive triggering (fixed intervals waste context), semantic-task mismatch (embeddings measure association, not relevance), and mathematical limits (embedding dimension constrains representable document sets). These require fundamentally different retrieval approaches, not tuning.
METEORA uses LLM-generated rationales with flagging instructions to select evidence, achieving 33% better accuracy with 50% fewer chunks than similarity re-ranking across legal, financial, and academic domains. The method also improves adversarial robustness substantially.
Twelve of fourteen models perform worse when constraints are removed, dropping up to 38.5 percentage points. Models appear to reason correctly by defaulting to harder options, not by actually evaluating constraints.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
- Can Large Language Models Reason and Optimize Under Constraints?
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- RAG-R1 : Incentivize the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism
- Does Thinking More always Help? Understanding Test-Time Scaling in Reasoning Models
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL