If news breaks and the AI quietly updates mid-study, how would you ever know which one changed its election answers?

This explores why measuring how AI election answers differ across users gets muddied when the world and the models both keep changing during the study. The clearest case in the corpus is MIT's Election LLM Observatory, which sent about 19,000 matched prompts to a dozen models and found real differences depending on who was asking and from where Where do AI election answers diverge across model and identity?. The catch is that the design shows *where* and *when* answers differ but not *why*. A shift could come from a deliberate policy change by the model provider, from breaking campaign news reaching the model through retrieval, from retrieval sources drifting over time, or from ordinary randomness. All four can produce the same pattern in the data.

Ordinary randomness is a bigger problem than it sounds. Work on prompt sensitivity finds that models give stable answers when they're confident and swing a lot when they aren't Does model confidence predict robustness to prompt changes?. Contested election topics are exactly where you'd expect low confidence. So some of the apparent difference by identity could just be a nervous model reacting to small changes in wording. Any study needs to know how much the answers vary with nothing else changed before it can say a difference is meaningful. The opposite also applies: separate research finds that very different models often converge on nearly identical answers because they share training data and alignment methods Do different AI models actually produce diverse outputs?. That means real divergence is more unusual than you might think, but it also means a single upstream change can move many models together. That looks like a coordinated response to an event when it isn't one.

Model updates also bring a less obvious problem: a model's political leanings aren't just a surface setting. Analysis of models' internal features finds big differences in how richly they represent political ideas, and models with deeper representations are harder to push in a new direction Can we measure how deeply models represent political ideology?. A version update could change that depth without anyone announcing it, so a study running over several months may end up comparing different models that share one name. There's a further measurement risk. Models can often tell internally when they're being tested, even if they rarely say so Do models know when they're being evaluated?. Systematic, templated probe prompts might get different answers than real voters' messy questions do.

Other parts of the corpus suggest ways to untangle this. The forecasting system Nexus does better by splitting numerical trends from event-driven context into separate stages rather than making one model handle both Can decomposing forecasting into stages unlock numerical and contextual reasoning?. Applied here, that would mean modelling the campaign-news calendar as its own explicit input instead of letting it leak into the results. Research on self-improvement makes a related point: reliable progress depends on fixed external reference points, such as frozen past model versions or outside judges Can models reliably improve themselves without external feedback?. For election monitoring, the equivalent is a pinned, version-locked model with retrieval switched off, run alongside the live one. That separates "the model changed" from "the world changed".

The corpus has only one study aimed directly at this question, so the methods above are carried over from other areas, not tested on elections. What the corpus does show is that a lot of divergence means little until you've separated out noise, events and updates.