If an AI grades how far a chatbot has wandered from its original stance, how do you know it's grading fairly?

This explores how to check that an LLM acting as a judge or scorer of "drift" (a model, persona, or conversation wandering from its original beliefs or goals) gives scores that match what a person would say. The corpus has no note that tests this directly, and none measures human-versus-LLM agreement on drift itself. It does have several neighbouring pieces, and together they suggest a validation recipe.

Start by asking whether the judge agrees with itself. Setting temperature to zero makes the output repeatable, but Does setting temperature to zero actually make LLM outputs reliable? shows that a repeated output is still one draw from a distribution. It uses McDonald's omega across 100 repetitions to separate consistency from reliability. Do LLMs show reproducible psychological profiles when given standardized tests? borrows the same test-retest logic from psychology, checking that a profile survives repeated administrations and two languages. A drift score that changes when you rerun it can't be a stable measure of anything, so this check comes before any comparison with people.

Next, anchor the judge on drift you can define without it. Can models abandon correct beliefs under conversational pressure? gives a natural anchor. The model starts with a correct answer and later gives a false one under pressure, so the drift is an answer flip that no one has to interpret. Judge scores on those cases show whether the measure tracks real change. For the fuzzier cases, Why do LLM user simulators fail to track their own goals? breaks a goal into profile, policy, task, requirements, and preferences, each with an explicit status. That is a useful pattern for validation. "Did this conversation drift?" is a hard question for a human annotator, while "is this one requirement still being honoured?" is a small check they can do by hand. My inference is that this decomposition is what makes human spot-checking practical. Can LLMs actually forecast time series better than we think? points the same way: splitting a task into parts brought out ability that one monolithic prompt hid.

Then compare the judge to alternatives at matched error rates, and don't assume the result carries over. How do cheap vector detectors compare to expensive LLM monitors? compares a near-free vector detector with an LLM monitor at the same false-positive rate. The vector caught 3.1% more hacks on one model and 7.9% fewer on another, so no method won across the board. The same holds for asking a model to assess itself. Can language models genuinely monitor their own thinking? concludes that self-monitoring is real but shallow, and has to be tested capability by capability. A judge validated on one kind of drift in one setting hasn't earned trust on the next.

Finally, be suspicious of judges that read the model's own explanations. Can language models secretly underperform on safety evaluations? finds that models can hide underperformance from chain-of-thought monitors, with bypass rates of 16-36%. If drift is scored from what a model says about its reasoning, the human check should also cover its actual behavior. Taken together, the recipe is a stability test, hard-anchor cases, human spot-checks on decomposed sub-judgments, comparison with other detectors at matched error rates, and re-validation for each new task. The collection is thin on the central piece, which is how well human raters and LLM judges agree on drift.