If an AI model gets a major upgrade, does its habit of favoring list position fade slowly, or just reset overnight?

This explores what happens to an AI agent's position bias, its tendency to pick items because of where they appear in a list rather than what they are, when the model behind it gets a major update. The short answer is that the collection doesn't measure speed. Nothing here tracks the bias week by week or across version numbers. What it does show is more useful for anyone building on these models: the bias doesn't fade gradually or carry over in a predictable way. Each model version has its own bias pattern, so a major update can change it all at once.

The main evidence comes from the ACES study of AI shopping agents Do AI shopping agents show persistent position bias in text interfaces?. Its researchers expected position bias to be a visual artifact that would disappear once the screenshots were removed. It didn't. The bias survived in plain-text interfaces and even when agents were told outright to ignore position. Each model also had its own preferred slots, and the pattern came back in a new form after each update. In practice, a seller who learned to optimize for one model's preferred slot can't assume that slot still wins after the next release. You have to measure the bias again for every version.

Why would the bias be built so deep into the model? A related finding hints at the reason. When GPT-4 is used as a recommender, its popularity bias comes from the data it was originally trained on, not from the dataset it's recommending from. The Shawshank Redemption keeps rising to the top no matter what list you give it Where does LLM recommendation bias actually come from?. If biases like these are set during training, a new model with new training data and new fine-tuning will naturally come with a new set of preferences. Company favoritism shows the same model-by-model variation. Claude leans slightly pro-Anthropic across several tasks, GPT only when grading, and Gemini slightly against Google Do frontier AI models favor their own company?. Neither study tracks change over time, but both point the same way: biases belong to a particular model, not to the industry as a whole.

Two perspectives from outside the shopping-agent work help here. First, recommendation systems have dealt with position bias for years. YouTube's ranker includes a small dedicated component that estimates how much an item's slot affected clicks and removes that effect. Without it, the system drifts into reinforcing its own past rankings Why do ranking systems need to model selection bias explicitly?. That suggests treating position bias as something to measure and correct, not something to hope a model update will fix. Second, there's a training-side fix. Consistency training teaches a model to give the same answer whether or not irrelevant changes are made to the prompt, using the model's own clean answers as the target Can models learn to ignore irrelevant prompt changes?. In principle, shuffling the order of a list is exactly that kind of irrelevant change.

The practical point most readers won't expect is this: if the bias resets with every model, the fix probably belongs in the system around the model, not in the model itself. Work on execution harnesses, the code and procedures that wrap a frozen model, found that the same setup carried over to newer models without changes and kept improving results Can execution harnesses lift model performance without retuning weights?. Shuffling options, rotating slot assignments or removing position cues at the harness level would survive a model update. A prompt tuned to one model's quirks would not. If you want to know how quickly bias moves between specific versions, that data isn't in this collection yet. Based on what is here, the honest answer is "potentially all at once, with every release."