Different AI shopping assistants each seem to favor different spots in a list — not one shared blind spot, but a different one for each.
Do all large language models show the same position bias in shopping tasks?
This explores whether AI shopping agents built on different language models all favor the same spots in a product list (say, the first item), or whether each model has its own pattern, and what that means for sellers and shoppers.
This explores whether different AI shopping agents share one position bias, such as always favoring the top listing, or whether each model leans its own way. The most direct evidence in the collection says no, they don't share one. The ACES study found that position bias in shopping agents is strong and **model-specific**: each model has its own preferred slots. The bias survived a text-only interface with no visual layout, and it survived explicit instructions to ignore position. The detail most people won't expect is that the pattern shifts after each model update. A merchant who tuned their listing placement for one version of an agent may find the tuning stops working on the next release Do AI shopping agents show persistent position bias in text interfaces?.
Why would models differ? One explanation is that these biases are set early. A causal study of cognitive biases found that models built on the same pretrained base model show similar bias patterns, whatever fine-tuning they get afterward. Instruction tuning only nudges biases that pretraining already planted Where do cognitive biases in language models come from?. That study measured cognitive biases, not shopping position, but it suggests a testable guess: agents may group by model family rather than by vendor or product. Two assistants built on the same base model could share position preferences, while models from different labs could diverge. There is also a possible low-level mechanism to explore. Researchers have found that a handful of unusually large internal values steer a model's attention toward particular tokens, regardless of the input Do hidden massive activations act as attention bias terms?. Whether this drives position bias in product lists is still unproven.
The opposite finding is also worth knowing. On a different kind of judgment, models can agree with each other in uncomfortable ways. Across 15,000 simulations, six LLMs recommended the same side of every business-strategy tension they were tested on. Yet simply swapping the order of the options moved their answers about 19%, more than changing the industry context did (11%) Do LLMs consistently favor the same strategic choices regardless of context?. So the picture is mixed. Different models can share preferences about *what* to recommend while each stays sensitive to *where* an option appears. When many people rely on the same few models, those shared tendencies spread across everyone using them Do large language models narrow human expression and thought?.
Why doesn't telling the agent to ignore position fix it? Other parts of the collection fit the ACES result. Models often ignore instructions in their context when habits learned in training are strong, and prompting alone usually can't override those habits Why do language models ignore information in their context?. Larger and instruction-tuned models follow user context even less often Do larger models follow stated beliefs less often?. Bias-reducing persona prompts tend to shift where the bias appears rather than remove it Can persona prompts actually reduce bias in language models?. A more promising fix happens during training: consistency training teaches a model to give the same answer when irrelevant details of the prompt change Can models learn to ignore irrelevant prompt changes?. In principle, shuffled product order is exactly that kind of irrelevant change.
One caveat: the collection has only one study that tests position bias directly in shopping. The idea that biases cluster by model family is inferred from research on other kinds of bias, not measured in shopping agents. The practical takeaway is still clear. If you sell through AI agents, there is no single "top slot" to win. Each model has its own preferred slots, and those preferences can change with every update.
Sources 9 notes
ACES found that AI agents exhibit strong, model-specific position biases even in headless text-only interfaces with explicit instructions to ignore position. The bias is a deep-seated model characteristic, not a superficial visual artifact, and recurs after each model update.
A causal experiment using random-seed variation and cross-tuning showed that models sharing a pretrained backbone exhibit similar bias patterns regardless of finetuning data. Biases are planted during pretraining and merely swayed by instruction tuning.
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
Across 15,000 simulations, six LLMs recommended the same strategic choice in every tension tested. Industry context shifted bias only 11%, while option order—a framing artifact—shifted results 19%, revealing that models recombine trend-coded vocabulary rather than analyze context.
LLMs mirror skewed slices of human experience shaped by training data regularities, and widespread reliance on identical models amplifies convergence. Co-writing studies show users unconsciously adopt model stances and framings.
Show all 9 sources
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Across 18 LLMs tested with EoBench, bigger models and instruction-tuned variants showed lower rates of context-following when users expressed beliefs that contradicted world knowledge. The effect suggests instruction-tuning strengthens reliance on parametric knowledge.
Across three models, persona conditioning makes models follow trait instructions but fails to eliminate underlying bias. Between-group sentiment gaps persist unchanged, showing prompts operate only at the output level.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMs
- Language models show human-like content effects on reasoning tasks
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- The Illusion of Debiasing: Persona Steering Redistributes Rather Than Reduces Bias in LLMs
- Consistency Training Helps Stop Sycophancy and Jailbreaks
- Vertical tacit collusion in AI-mediated markets
- When Persona Attributes Improve Population Alignment in Large Language Models
- Massive Activations in Large Language Models