Can scarcity of solutions protect benchmarks from data contamination?

Benchmark builders usually treat missing ground truth as a defect. ExploitGym's limitations paragraph concedes it: "our benchmark lacks ground-truth exploits for every task due to the extreme difficulty of exploitation." Then it turns the defect around: "at the same time, this helps mitigate data-contamination concerns, since complete solutions are not broadly available." The excerpt's wording is ambiguous about whether some tasks have known exploits and others do not, or none do; either way, the contamination argument is the same.

The mechanism is simple. Contamination happens when the answer to a test item appears in training data, so the model recalls it instead of reasoning. If complete working exploits for these vulnerabilities are not broadly published, there is little for a model to memorize, and a success has to come from constructing something. Difficulty becomes the contamination defense: the task is protected because it is hard enough that nobody has widely shared a solution.

This is a different defense from the one the vault has recorded so far. Can live benchmarks prevent data contamination in prediction tasks? defends by time: the answer does not exist yet when the model makes its prediction. Difficulty-based protection defends by scarcity: the answer exists in principle but is not in circulation. It is aimed at the same problem shown in Does RLVR success on math benchmarks reflect genuine reasoning improvement? and How much of LLM few-shot ability comes from training data?, where memorization inflated apparent capability.

Two limits apply. First, scarcity is not permanent: solutions can be published after the benchmark is, or a model could be trained on newly public exploits, so this protection may erode. That is my inference; the excerpt does not discuss it. Second, the same missing ground truth has a cost the authors also name: without a known solution, a failure cannot be distinguished from a task that was impossible. That cost is drawn out in the tension between these two properties, which the vault should track.