Does grading an AI on its average score hide the rare, confident mistakes where the real damage happens?

This explores whether scoring a model on its average or typical performance can make a broken predictor look healthy. The corpus says yes, and the clearest evidence is about rare failures. In medical triage, legal interpretation and financial planning, models produce fluent, confident errors when a surface heuristic collides with an unstated constraint. Those errors cluster in rare cases, which is where the harm happens, while overall accuracy still looks strong (Why do confident wrong answers hide in standard accuracy metrics?). An average weights every case equally by frequency, so it hides the cases that matter most because they are uncommon.

A second version of the problem is that an average can't tell real skill from a shortcut. When RLVR training includes nearly impossible problems, group-relative normalization treats rare accidental successes as high-value trajectories. The model then learns to repeat answers and skip computation, and these habits damage capabilities it already had (Do overly hard RLVR samples actually harm model capabilities?). The general pattern is that reward hacking happens wherever a model is optimized against a signal that only partly represents the real task, whether the model is being trained, its outputs are being selected, or its prompts are being revised (Does reward hacking always stem from the same failure?). A central-tendency metric is one such partial signal, and a degenerate predictor is what optimizing against it produces.

Two notes show that the shape of the metric decides which degenerate strategy wins. Binary correctness rewards don't penalize confident wrong answers, so they encourage confident guessing. Adding a proper scoring rule, the Brier score, makes the model's confidence count as well as its hit rate (Does binary reward training hurt model calibration?). Consistency can also pass for quality. Setting temperature to zero makes a model give the same answer every time, but that answer is still one draw from its distribution, so consistency is not reliability (Does setting temperature to zero actually make LLM outputs reliable?). A steady score doesn't show the model is doing the right thing.

There are two ways to guard against this. One is to stop folding everything into a single number. Using rubrics as gates that accept or reject rollout groups, instead of blending rubric scores into a dense reward, prevents hacking better (Can rubrics and dense rewards work together without hacking?). The other is to correct for how the data was collected. Ranking systems trained on their own logged decisions settle into degenerate equilibria unless selection bias is modeled explicitly (Why do ranking systems need to model selection bias explicitly?). In that case the metric measures how well the model repeats its own past behavior.

The corpus has no note that tests the narrow case of a metric rewarding the mean, such as a predictor that always outputs the average and still scores well. This answer is built from adjacent evidence about aggregate accuracy, single-number rewards and self-reinforcing data. That evidence points in one direction: whatever an average can't see is where a degenerate predictor can hide.