Some AI tests stay un-cheatable because the answers haven't happened yet; others because almost nobody has written them down. How do those defenses differ?
How does contamination protection by time differ from protection by scarcity?
This explores two different ways a benchmark can keep its answers out of a model's training data: by asking about things that haven't happened yet (time), or by asking for things almost nobody has published (scarcity).
This explores two different ways a benchmark can keep its answers out of a model's training data: asking about things that haven't happened yet (time), or asking for things almost nobody has published (scarcity). The corpus has one note on each, and together they show that the two defenses fail in different ways.
Protection by time works because the answer doesn't exist yet. A live benchmark like FutureX keeps collecting fresh questions from trusted sources and grades models against what actually happens. No model can have memorized an outcome that hadn't occurred when it was trained, so the protection comes from the benchmark being live rather than retroactive Can live benchmarks prevent data contamination in prediction tasks?. A retroactive test on past events can't promise this, because the answers are already somewhere on the internet.
Protection by scarcity works because the answer exists but is rare. ExploitGym's tasks lack ground-truth exploits, and complete working exploits are not widely published. A model therefore has to build a solution instead of recalling one Can scarcity of solutions protect benchmarks from data contamination?. The answer is hidden, not absent, and that is why this protection is fragile. The note says it may erode once solutions get published after the benchmark, and after that the scarcity is gone for good.
The practical difference is in how each defense holds up over time. Time protection is renewable but costs ongoing effort: a live benchmark stays clean only while someone keeps feeding it new questions and checking real outcomes. Scarcity is free to start with but only decays. It depends on what the world hasn't published yet, and every new write-up or leaked solution shrinks it. The first defense is a process you keep running, and the second is a resource you use up.
The corpus doesn't compare the two head to head, so this contrast is my synthesis of two separate notes. It does suggest a design rule. Scarcity can protect a benchmark on day one, but time-based freshness is the sturdier defense if you want it to stay clean as models and the public record grow.
Sources 2 notes
FutureX demonstrates that continuously collecting questions from trusted sources and checking actual outcomes creates a contamination-free benchmark. Being live—not retroactive—is the key defense against answers leaking into training data.
ExploitGym's missing ground-truth exploits reduce data contamination risk because complete working exploits are not widely published. Models must construct solutions rather than recall them, though this protection may erode as solutions are published post-benchmark.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Task Contamination: Language Models May Not Be Few-Shot Anymore
- An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
- FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
- A Causal Model for Locating and Unlocking Sandbagging in Model Organisms
- Hack-Verifiable Terminal Bench: Evaluating Reward Hacking in Terminal Tasks
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts