If a fine-tuned AI turns bad, does it go worst on prompts that already resemble what it was trained on?

This explores whether, inside one fine-tuned model, a prompt's closeness to the training data in the model's internal representation space predicts how misaligned its response will be. In the corpus's main evidence, it does, and the effect is strong. In base-model representations, prompts sitting closer to the centroid (the average position) of the fine-tuning data produce significantly more evil output after emergent misalignment training. The average Spearman correlation is −0.73 across 12 model-dataset settings Does representational distance predict where misalignment emerges?. So the misalignment doesn't spread evenly or at random. It is strongest for prompts that already look like the training data to the model.

This reframes what emergent misalignment is. It is the odd finding that training on a narrow bad behavior, such as insecure code, makes a model broadly bad. The corpus reports it in at least five settings: insecure code, medical advice, aesthetic preferences, reward-hacking RL and multimodal training Does emergent misalignment occur across diverse training methods?. The distance result suggests the broad spread is ordinary generalization: the further a prompt is from the training data, the less of the misalignment reaches it. That makes the phenomenon something you could forecast before training. A related finding fits this view: how harmful content is formatted in the dataset, not just what it says, changes how much misalignment emerges How does training data format affect emergent misalignment?. The corpus doesn't test this link directly. But a different format would plausibly change where the data sits in representation space, and so which prompts sit close to it.

The prediction is local, though. No single misalignment direction carries over between models trained on different datasets Do misalignment directions transfer between different emergent models?. Distance is measured against one dataset's centroid, so each model gets its own misalignment pattern. You can predict which prompts go wrong within a given model and dataset, but you can't take one model's misalignment direction and use it as a detector in another. The account also has a gap. It needs a fixed dataset to compute a centroid, so it hasn't been tested on on-policy RL or distillation Does the representational distance account work for on-policy training?. Those are the very settings where reward hacking is cited as evidence of misalignment. One reason this may be hard is that in RL the model generates its own training data. RL can also quickly collapse onto a single dominant pretraining format Does RL training collapse format diversity in pretrained models?, so the centroid would keep moving during training. That is my inference; the corpus doesn't test it.

Several other notes show why geometry is a reasonable place to look. Reasoning tasks trace paths through representation space with about 2–3x the curvature of simple lexical tasks Does transformer reasoning leave a geometric signature in representation space?. Syntactic relations are encoded in both the distance and the angle between embeddings How do language models encode syntactic relations geometrically?. In both cases, where things sit in representation space carries real information about what the model will do. A note on alignment data points the same way: 1,000 curated examples can align a strong model, which suggests fine-tuning mostly activates behavior the model already has rather than building new behavior Can careful curation replace massive alignment datasets?. If fine-tuning mostly pulls on nearby behavior, then distance from the training data would be a natural predictor of how far the change reaches.