Models that nail average accuracy can still flip the wrong decisions in the close calls that matter most — so when does good data-fitting actually mean good choices?

This explores when a model that predicts well on average actually helps someone choose a better action, which is a different thing from being accurate. The most direct answer in the corpus is narrow but sharp. Why do accurate predictions lead to poor decisions? works out the exact conditions under which a predictive model supports optimal decisions. The main point is that average accuracy is the wrong thing to measure. A model can be right most of the time and still be wrong in the few situations where the decision flips, such as the borderline patient, the marginal loan applicant, or the case right at a policy threshold. Data fit helps decisions only when the model's accuracy lands in those decision-critical states. Fitting the data well overall doesn't guarantee that.

The rest of the corpus suggests this isn't a quirk of forecasting. It's a general pattern, and it shows up under other names. In AI training it's called Goodhart's Law: once you optimize a measurable stand-in for what you actually care about, the system learns to satisfy the stand-in. How vulnerable is AI training to Goodhart's Law? traces this through reward hacking, sycophancy and benchmark contamination, and concludes there is no complete fix, only partial mitigations. Seen this way, 'data fit' is just another proxy. It supports good policy only to the extent that the errors it allows fall where they don't matter.

There's a second, less obvious condition: the prediction must not change the world it predicts. Once a model's forecast drives a policy, the policy changes future data. Do models recognize their own outputs as actions shaping future inputs? shows a version of this inside language models: after post-training, models start treating their own outputs as actions that shape what they see next, rather than as passive predictions. That paper is about LLMs, not policy models, but the lesson carries over. A model trained purely to fit past data assumes it is a spectator. Decision-support models are participants, so fitting history is necessary but not enough.

So what actually closes the gap? Two notes point the same way: you need a signal that checks what matters, not more fitting. When can weak models match strong model performance? finds that generating many candidate answers widens coverage but can't pick out the right one without an external check such as a test or a proof. Decision-making works similarly: a model needs feedback tied to decision outcomes, not just to prediction error. What predicts success in ultra-long-horizon agent tasks? adds that repeatedly measuring, adjusting and re-measuring beats initial quality. And Can decomposing forecasting into stages unlock numerical and contextual reasoning? suggests that separating number-crunching extrapolation from reasoning about events and context gives forecasts that hold up better when circumstances shift, which is often exactly when policy decisions matter most.

Here's the useful reframe: 'better' depends on who uses the output. Does teacher-refined data always improve student model performance? finds that objectively higher-quality training data can hurt a student model when it's beyond what that model can absorb. Predictions work the same way. The best-fitting forecast isn't automatically the most useful one; what counts is whether it's accurate where the decision-maker's choice actually turns. One caveat: the corpus has one formal paper on prediction-for-decisions and doesn't cover applied policy settings (public health, economics, criminal justice), so read the parallels above as useful analogies, not settled evidence.