Could a simple 'personality dial' inside an AI reveal, before training, which datasets will make it more sycophantic?
How do trait vectors in activation space predict which datasets cause personality shifts?
This explores whether the internal directions that correspond to traits like sycophancy can flag, before training, which fine-tuning datasets will shift a model's personality.
This explores whether the internal directions that correspond to traits like sycophancy can flag, before training, which fine-tuning datasets will shift a model's personality. The corpus has one note that answers this directly. Researchers found that traits such as sycophancy and hallucination correspond to linear directions in a model's activation space, and that these "persona vectors" predict finetuning-induced personality shifts before they occur Can we track and steer personality shifts during model finetuning?. A trait becomes a direction you can measure, so a dataset can be checked against it. The same directions can also steer training away from unwanted trait changes as it runs. The retrieved summary doesn't spell out the exact scoring recipe, so treat "predicts" as the headline claim and the mechanics as something to read in the note itself.
A geometric predictor matters because reading the dataset often doesn't work. Models can pick up behavioral traits from filtered data that has no semantic relationship to the trait, and the effect survives rigorous filtering Can language models transmit hidden behavioral traits through unrelated data?. It is also model-specific and fails across different architectures, which suggests the signal is a statistical signature rather than meaning. A content check would miss that kind of dataset. A vector measures what the data does inside a particular model, not what it says. That would also explain why the prediction is tied to the model whose activation space you mapped.
Other notes suggest personality has structure that can be located. Mapping hundreds of character archetypes gives a low-dimensional persona space. Its leading axis measures how far the model sits from its default Assistant, and emotional or meta-reflective conversations push it away along that axis in predictable ways How stable is the trained Assistant personality in language models?. Capping activation along that axis reduces harmful shifts without hurting capabilities. Fine-tuning on Big Five traits also produced emojis that were never in the training data, and the change traced to specific deepest-layer neurons that became trait-specialized Do personality traits activate hidden emoji patterns in language models?. Fine-tuning changes reach well past the surface of the training text, so predicting them from the text alone is hard. The change also appears to be localized.
The weights are probably the right place to watch because prompting is a weak lever. Most open models resist prompted personalities and hold onto their trained ENFJ-like defaults Can open language models adopt different personalities through prompting?. That resistance doesn't improve with model scale Why do AI personas default to the same personality type?. Adapters that modify every transformer layer bypass prompt resistance entirely with under 0.1% extra parameters Can we control personality in language models without prompting?. Taken together, personality is shaped in the model's internals, and a trait vector is a way to read those internals ahead of time. The corpus doesn't say how reliable these predictions are across many traits or how they hold up against subliminal transfer, so those are open questions.