Should an AI predict the future from what it already knows, or go look things up first?
Do base models or search agents win at open-ended prediction tasks?
This explores whether a language model answering from what it learned in training does better at open-ended prediction (like forecasting real-world events) than an agent that searches for evidence before answering.
This explores whether a language model answering from what it learned in training does better at open-ended prediction than an agent that searches for evidence first. The corpus has no head-to-head test of the two on the same questions. What it does have suggests they aren't really rivals: the searcher's ceiling depends on the model underneath it.
The closest evidence is a forecasting system that pairs a language model with retrieval Can retrieval-augmented language models forecast like human experts?. On real questions published after the models' training cutoffs, it came close to competitive human forecasters and sometimes beat human crowds. Those questions were about events no model could have memorized, so the search step was doing real work. The paper also found that newer model generations improved the forecasts with no forecasting-specific tuning. Progress in the base model flows straight into the searching system.
The case for the model alone is stronger than it first looks. Base models already hold more reasoning ability than they show, and post-training mostly selects it rather than creating it Do base models already contain hidden reasoning ability?. The line between 'knows' and 'looks up' is also blurry: 14B models that simulate a search engine from their own internal knowledge matched or beat real search engines as training environments for agents Can LLMs replace search engines during agent training?. That result is about training agents, not predicting the future. It shows that a model's memory is richer than assumed, but it doesn't show the model can see past its cutoff.
Here is where I'd expect a model working alone to struggle. LLMs are probability machines, and they fail more often when the correct answer is a low-probability one, even on logically simple tasks Can we predict where language models will fail?. Applying that to forecasting is my extrapolation, not the paper's claim. Surprising outcomes are what a model's priors underrate, and fresh evidence is what could flag them. Search has its own failure mode, though. Embedding-based retrieval can blur together similar entities, and an agent that greps raw text directly beat it on multi-hop questions with precise entity constraints Can direct corpus search beat embedding-based retrieval?. How an agent searches decides what it finds.
The two sides also feed each other. What search agents read, and especially the distractors they read but don't cite, can be mined as process rewards to train better reasoning Can search agent behavior yield reliable process rewards for reasoning?. So for events after the training cutoff, model plus search is the better bet. The more interesting story is that the two improve together. What's missing is a controlled test of a pure model against a searching agent on identical open-ended questions.
Sources 6 notes
A retrieval-augmented LM system achieved near-parity with competitive human forecasters on real forecasting questions published after model training cutoffs, sometimes surpassing human crowds. Newer model generations naturally improved forecasting without domain-specific tuning.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
ZeroSearch and SSRL demonstrate that LLMs can generate relevant documents and search results from internal knowledge, with 14B simulators matching or exceeding real search engines. Curriculum degradation and test-time scaling optimize this approach for training without API costs.
By framing LLMs as autoregressive probability machines, researchers predicted tasks with low-probability target responses would be systematically harder, even when logically simple. Experiments confirmed predictions like backwards alphabet and letter counting.
GrepSeek trains agents to retrieve via executable shell commands over raw text, achieving better multi-hop performance on entity-constrained queries than dense embeddings. The approach scaffolds unstable search mechanics with supervised trajectories, then refines task-oriented behavior through reinforcement learning.
Show all 6 sources
LongTraceRL mines entity-level reasoning signals from what search agents read but don't cite—the hardest distractors—and applies rubric rewards only to correct answers, structurally blocking reward fabrication while capturing intermediate reasoning quality.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses
- ZeroSearch: Incentivize the Search Capability of LLMs without Searching
- Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks
- GrepSeek: Training Search Agents for Direct Corpus Interaction
- Approaching Human-Level Forecasting with Language Models
- Eliciting Reasoning in Language Models with Cognitive Tools
- LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards
- Base Models Know How to Reason, Thinking Models Learn When