Can retrieval-augmented language models forecast like human experts?

Judgmental forecasting — assigning probabilities to future events from judgment, domain knowledge, and reasoning under distributional shift — is where humans have historically beaten statistical models, and where competitive forecasters set a high bar. This work builds a retrieval-augmented LM system that searches for relevant information, generates forecasts, and aggregates predictions, evaluated on a large dataset of questions from real forecasting competitions, tested only on questions published after the models' knowledge cutoffs (so the answers can't be memorized). The result: on average the system nears the crowd aggregate of competitive forecasters, and in some settings surpasses it — the first ML system to forecast at near-human levels. Two design pieces matter: a novel LM-driven retrieval mechanism that decides what to source and how to evaluate relevance, and a self-supervised finetuning method to generate reasonings with accurate predictions.

The keeper is twofold: scalable near-human forecasting is now feasible, and newer model generations forecast better naturally — capability rises with the base model without forecasting-specific tricks.

This is the foundational human-level-forecasting result the vault's forecasting cluster builds on. It underpins Can language models beat human venture capital experts? (VCBench) and Can LLMs actually forecast time series better than we think?, and pairs with the contamination-defense of Can live benchmarks prevent contamination in prediction tasks? (post-cutoff test sets are the same leak-proofing principle).