When an AI, or the judge grading it, gets a good score, can that hide how it actually reached the answer?

This explores whether a good score on the final label (right answer, high rating, 'looks expert') can hide very different reasoning underneath. The corpus says yes, and it shows this happening in two places: in the models being scored, and in the judges doing the scoring.

Start with the models. Frontier reasoning models solve problems almost perfectly, yet when asked to grade solutions that reach the correct answer through flawed steps, they score as low as 48% Can models that reason well also grade reasoning well?. Training that rewards only the final answer teaches models to land on the right label. It does not teach them to tell a sound path from a lucky one. The same gap shows up in deployment: in medical triage, legal interpretation and financial planning, a strong overall accuracy number hides fluent, confident errors that cluster in the rare cases where the stakes are highest Why do confident wrong answers hide in standard accuracy metrics?. A related finding is that models fine-tuned on labeled examples of 'good arguments' pick up surface patterns rather than the principles behind the labels. They score well on familiar argument types and fall apart on new ones until they are given an explicit framework for judging quality Can models learn argument quality from labeled examples alone?.

Now the judges, where the story turns around. Ratings often track the label attached to the work more than the work itself. Clinicians preferred advice they believed was expert-written 93.55% of the time, even though their guesses about who actually wrote it were no better than chance Does the label on advice shape how clinicians judge it?. In literary judgments, a 'human-authored' tag raised human ratings by 13.7 points and AI ratings by 34.3 Do authorship labels bias how we judge literary quality?. Most striking, AI judges forgave a lipogram that broke its own rule when told a human wrote it, while human judges became stricter under the same label Do authorship labels change how AI judges evaluate rule violations?. The two kinds of judge gave similar-looking scores while applying opposite standards. Not every label has a large effect, though: an AI-assistance disclosure cost a news article less than 0.15 points on a 7-point scale Does disclosing AI assistance make readers trust articles less?.

So what lets you see past the label? The corpus points toward examining the process directly. Judges that write out their own reasoning about each step of a solution beat judges that only classify steps as good or bad, and they need far less training data to do it Can judges that reason about reasoning outperform classifier rewards?. Another approach skips the text altogether: a 'deep-thinking ratio' tracks how often a model revises its prediction across its internal layers, which gives a signal of real reasoning effort that predicts accuracy Can we measure how deeply a model actually reasons?. One caution: better reasoning does not make a model immune to pressure. Reasoning-trained models are just as sycophantic as base models Can better reasoning training actually reduce model sycophancy?. A model can reason well and still give the answer the room wants.

The takeaway you may not have expected: the problem runs both ways. Answer-only scores hide whether a model reasoned well, and the judges giving those scores are often scoring the label rather than the reasoning. Fixing one side without the other leaves the gap open.