An AI can ace every test question and still be a tangled mess inside — can scores catch that?
Do perfect accuracy scores hide broken internal representations?
This explores whether a model can score perfectly on a test while the way it organizes what it knows is broken underneath, and whether ordinary accuracy metrics can detect that.
This explores whether a model can score perfectly on a test while its internal organization is broken underneath, and whether accuracy numbers can tell. The corpus says yes, and the sharpest evidence is the Fractured Entangled Representation hypothesis. Networks trained with standard SGD can produce identical outputs on every input, yet their internal structure is radically different from that of networks evolved for the same task. Nudging the weights shows a tangled, fractured mess where the evolved networks have clean structure (Can identical outputs hide broken internal representations?). The unsettling part is that the benchmark can't see it. A perfect score tells you about the inputs you tested, not about the machinery that produced the answers (Can AI pass every test while understanding nothing?).
The damage shows up when you ask the model to do something new. Fractured representations block transfer to unfamiliar contexts and block creative recombination of what the model already knows. So the practical test of whether a perfect score is hiding something is not to run more of the same questions. It is to see whether the model can carry its skill somewhere it hasn't been.
The same blind spot appears at the other end of the pipeline, in deployment, though the mechanism is different. In medical triage, legal interpretation and financial planning, models give fluent, confident answers that follow a surface heuristic and miss an unstated constraint. These errors cluster in rare cases, which is exactly where harm happens, and because they're rare the aggregate accuracy still looks strong (Why do confident wrong answers hide in standard accuracy metrics?). This isn't a representation problem. It's an averaging problem, but the lesson is the same: one number compresses away the part you care about.
Two more cases show how a good-looking score can mislead. Models fine-tuned to imitate ChatGPT fooled human evaluators with confident, polished style while closing none of the capability gap on factuality or new tasks (Can imitating ChatGPT fool evaluators into thinking models improved?). Multimodal zero-shot results look like general understanding, but performance tracks how often a concept appeared in pretraining, and each linear gain costs exponentially more data (Does multimodal zero-shot performance actually generalize or interpolate?). Underneath all of these is the pattern named in the reward-hacking work: optimizing against a signal that only partly represents the real task will satisfy the signal without satisfying the task (Does reward hacking always stem from the same failure?). A test score is one of those signals.
Sources 6 notes
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
The Fractured Entangled Representation hypothesis shows that SGD-trained networks can produce identical outputs across all inputs while maintaining radically different internal representations. Standard benchmarks cannot detect this structural difference.
Medical triage, legal interpretation, and financial planning show a consistent pattern: surface heuristics conflict with unstated constraints, producing fluent confident errors that concentrate in rare cases where harm occurs. Aggregate accuracy masks these failures because overall performance looks strong.
Imitation models fool human evaluators by mimicking ChatGPT's confident, fluent style while failing to improve factuality or generalization on novel tasks. The ceiling is set by base model capability, not fine-tuning method—better fundamentals, not shortcuts, drive real improvement.
Across 34 models and 5 datasets, multimodal models require exponentially more pretraining data for linear performance gains on downstream tasks. Performance correlates with how often test concepts appeared during pretraining, not genuine generalization ability.
Show all 6 sources
Reward hacking arises during weight training, output selection, and prompt revision through a shared failure: optimization against signals that incompletely represent the actual task. The substrate matters less than the misalignment between the scoring function and ground truth.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
- No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance
- The False Promise of Imitating Proprietary LLMs
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
- Inducing Emergent Misalignment from Reward Hacks with Iterative DPO