What privacy-preserving evaluation methods best capture real-world forecasting ability?
This reads the question as two threads the corpus keeps separate — 'real-world forecasting ability' (how do you measure genuine prediction, not memorized answers) and 'privacy-preserving evaluation' (how do you measure without leaking the data involved) — and asks where they meet.
This explores the meeting point of two problems the corpus treats apart: keeping a forecasting test honest, and keeping it private. The honest-test problem turns out to dominate the literature here, and its core method is contamination control. The strongest signal is that real-world forecasting can only be measured on questions whose answers didn't exist when the model was trained — live, open-ended benchmarks that pull questions from many sources and verify actual outcomes after the fact Can live benchmarks prevent contamination in prediction tasks?. The same logic drives retrieval-augmented forecasting evaluated only on questions published after the training cutoff, which reached near-parity with competitive human forecasters Can retrieval-augmented language models forecast like human experts?. The shared insight: if a benchmark can leak the future into the model, it measures recall, not foresight.
What's surprising is how close temporal-leakage prevention sits to actual privacy preservation — they're the same shape of problem. TiMoE makes this literal by training experts on disjoint time slices and masking any expert whose window postdates the query, turning 'don't peek at the future' into an architectural guarantee rather than a hope Can routing mask future experts to prevent knowledge leakage?. That's a contamination firewall that doubles as an information-access control — the same machinery you'd want if 'future knowledge' were instead 'private knowledge.'
On the genuinely private-data side, the corpus offers a sharper warning than a method. Reasoning traces leak private user data mostly by direct recollection — the model materializes sensitive details as cognitive scaffolding, longer chains leak more, and scrubbing traces afterward degrades the very capability you were testing Do reasoning traces actually expose private user data?. So any forecasting evaluation that rewards long agentic reasoning (which the live benchmarks show is exactly what hard forecasting needs) is also maximizing the surface for private-data leakage. The cleanest answer the corpus has to this tension is deployment-level: keep the data and the model local. The LLEAP work rated over a thousand therapy sessions with strong psychometric validity using an 8B model run on-premises, so sensitive transcripts never left local storage Can local language models rate therapy engagement reliably? — a privacy-by-architecture pattern directly portable to forecasting on sensitive data.
There's also a quieter, deeper point about what 'real-world' even means for forecasting. Real-world prediction is forecasting under information asymmetry — you don't see what the other agents see. LLMs look socially competent in 'omniscient' simulations where one model secretly controls everyone, and fail systematically once agents hold genuinely private information Why do LLMs fail when simulating agents with private information?. An evaluation that doesn't withhold information from the model is measuring an easier, omniscient task — meaning privacy-preserving design isn't just an ethics constraint here, it's part of what makes the test realistic.
If you're building such an evaluation, the corpus points to a stack rather than a single method: live, post-cutoff, outcome-verified questions for honesty Can live benchmarks prevent contamination in prediction tasks?; local or causally-gated model access so neither future nor private data leaks in Can routing mask future experts to prevent knowledge leakage? Can local language models rate therapy engagement reliably?; and decomposed, workflow-structured reasoning so you can score the forecast without dumping every sensitive detail into one long trace Can decomposing forecasting into stages unlock numerical and contextual reasoning? Can LLMs actually forecast time series better than we think?. One caveat worth knowing: no paper here is purpose-built as a 'privacy-preserving forecasting benchmark' — the answer is assembled laterally, and the field's bar for raw forecasting is sometimes low enough that even untuned models clear modest human experts Can language models beat human venture capital experts?, which is its own reason to evaluate carefully.
Sources 9 notes
FutureX, a live benchmark collecting questions from 195 sources and verifying real outcomes, shows that base models handle easy predictions but hard open-ended forecasting demands search-and-reasoning agents. This proves forecasting is an agentic capability, not a base-model strength.
A retrieval-augmented LM system achieved near-parity with competitive human forecasters on real forecasting questions published after model training cutoffs, sometimes surpassing human crowds. Newer model generations naturally improved forecasting without domain-specific tuning.
TiMoE pre-trains experts on disjoint two-year slices and masks experts whose windows postdate the query, cutting future-knowledge errors by ~15% while guaranteeing strict causal validity. This shows temporal grounding can be an architectural property, not just a retrieval patch.
74.8% of privacy leaks in language model reasoning traces result from models materializing sensitive user data during thought processes. Longer reasoning chains amplify leakage, and anonymizing traces post-hoc degrades model utility, suggesting private data functions as cognitive scaffolding.
LLEAP achieved reliability (omega=0.953) and valid correlations with motivation, effort, and symptom outcomes using Llama 3.1 8B to rate 1,131 therapy sessions, while keeping data locally stored.
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
Nexus outperforms pure TSFM and LLM baselines on real-world datasets by decomposing forecasting into contextualization, dual-resolution macro/micro outlook, and synthesis stages. Separating numerical extrapolation from event-driven contextual reasoning avoids forcing one model to handle both simultaneously.
LLMs have stronger intrinsic forecasting ability than recognized, but only when workflows separate numerical reasoning from contextual reasoning. Monolithic prompting obscures this capability; structured decomposition surfaces it.
VCBench shows several LLMs exceed human baselines in founder-success prediction, with DeepSeek-V3 achieving 6× market-index precision. In sparse-signal forecasting where experts only modestly beat chance, even raw LLM capability suffices to clear the human bar.