What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commerce

Online marketplaces will be transformed by autonomous AI agents acting on behalf of consumers. Rather than humans browsing and clicking, AI agents can parse webpages or leverage APIs to view, evaluate and choose products. We investigate the behavior of AI agents using ACES, a provider-agnostic framework for auditing agent decision-making. We reveal that agents can exhibit choice homogeneity, often concentrating demand on a few “modal” products while ignoring others entirely. Yet, these preferences are unstable: model updates can drastically reshuffle market shares. Furthermore, randomized trials show that while agents have improved over time on simple tasks with a clearly identified best choice, they exhibit strong position biases—varying across providers and model versions, and persisting even in text-only ”headless” interfaces—undermining any universal notion of a “top” rank. Agents also consistently penalize sponsored tags while rewarding platform endorsements, and sensitivities to price, ratings, and reviews vary sharply across models. Finally, we demonstrate that sellers can respond: a seller-side agent making simple, query-conditional description tweaks can drive significant gains in market share. These findings reveal that agentic markets are volatile and fundamentally different from human-centric commerce, highlighting the need for continuous auditing and raising questions for platform design, seller strategy and regulation.

Keywords: AI shopping agents, agentic e-commerce, platform design, marketplaces, sponsored ads, product ranking, recommendation systems, Vision Language Models

Introduction. In a significant structural change for e-commerce, AI tools for shopping assistance have recently become available to consumers, and are quickly gaining adoption [13]. Industry leaders are already signaling strategic disruption; Walmart’s CTO urges preparedness for third-party agents that circumvent traditional merchandising and advertising cues.1 In particular, autonomous AI agents offer the capability to take over the shopping journey entirely including product selection.

A new generation of “computer-use” systems—such as Agent mode in OpenAI’s Atlas browser or Google’s Project Mariner which can automate grocery orders, fill forms, and book flights— promises to replace manual browsing with autonomous procurement. With such tools becoming widely available, there is a need for a principled framework to study, evaluate and monitor these systems.

AI tools have the potential to significantly reduce search frictions by systematically scanning broader assortments than humanly possible. Yet, this raises a critical question: are AI agents the perfectly rational optimizers that economic theory has long envisioned, or are they a new species of decision-maker with their own unique, exploitable biases? If the latter, do these tools dilute the influence of classical monetization levers, such as product rankings and sponsored ads?

Parallel questions arise on the seller side: is there scope for “AI-SEO” where sellers tune listings to anticipate buyer-agent behavior? The emergence of these tools foreshadows a reconfigured e-commerce ecosystem, raising novel design, strategy, and regulatory challenges. With this shift upon us, we ask the fundamental question: what is our AI agent buying—and why? More concretely, we investigate this through various layers of inquiry.

1. Rational Behavior. Do agents satisfy basic instruction following and simple economic dominance tests? And how are such capabilities evolving over time?

2. Product Market Shares. What market shares emerge when purchases are fully mediated by AI agents, and how do they vary across model providers and generations?

3. Choice Behavior and Biases. How do AI agents respond to observable attributes (price, rating, reviews, text) and platform levers (position, promotions, sponsorship)? And to what extent are such responses stable across providers and model generations?

4. Interaction between Buyer and Seller Agents. How might outcomes change when sellers and/or marketplace platforms deploy their own optimizing AI agents?

The first question concerns minimum viability: without elementary rationality, higher-order design or policy levers may be moot (cf. the emphasis on baseline competence by Visa CEO in 1https://www.wsj.com/articles/walmart-is-preparing-to-welcome-its-next-customer-the-ai-shopping-agent- 6659ef18 an interview to Bloomberg2). The second and third probe “in-the-wild” behaviors. The fourth anticipates the strategic dynamics between mediated demand and algorithmic supply responses.

To investigate the aforementioned research questions, we construct a controllable agent-platform sandbox dubbed ACES and conduct randomized experiments that (i) test for instruction following and basic rationality, (ii) reveal market shares, (iii) cleanly isolate and quantify the effects of listing characteristics and platform tags on agent decision making, and (iv) understand the impact of a strategic response by a seller who optimizes their product description. Our main contributions

Related work. Our work connects to a rich body of literature spanning computer-use agents, e-commerce systems, recommendation and ranking algorithms, behavioral biases in human decision-making, and recent advances in multi-modal AI agents. Across these threads, our contribution is to provide a controlled, visual, and interactive sandbox that zooms in on the choice step (which product to buy) for AI shopping agents, and measures rationality and choice behavior via randomized page layouts, attributes, and badges. Although ACES resembles prior computer-use testbeds such as WebArena and VisualWebArena, our emphasis is different: rather than evaluating end-to-end web navigation, we focus on a single critical step—selecting which product to buy—to obtain a finegrained view of agent choice behavior in shopping contexts [26, 50]. While most benchmarks for AI agents evaluate end-to-end task completion, our contribution is to isolate the critical choice step in agent shopping through the controllable ACES sandbox. This facilitates the first agentcentric, causal evaluation of how AI shoppers respond to product attributes and platform levers like product positions and promotional badges, complementing existing studies on human consumers [29, 41].

Computer-Use Agents and Benchmarks. A fast-growing body of work builds web and OS- interacting agents evaluated in realistic environments. WebArena introduces reproducible, longhorizon web tasks over functional sites (e.g., e-commerce, forums), with baseline agents achieving low end-to-end success relative to humans [50]. VisualWebArena extends to multimodal interfaces with images and forms [26]. Mind2Web targets generalist web agents across 137 real websites [10], and follow-ups examine grounding for GPT-4V as a generalist web agent [49]. Beyond the browser, OSWorld provides a real-computer environment (Ubuntu, Windows, macOS) and documents large headroom between SOTA agents and humans [44]; Windows Agent Arena and AndroidWorld further expand to Windows and Android ecosystems [5, 31]. Work studying agent–computer interfaces shows how UI design and tool affordances affect agent success (e.g., SWE-agent) [46]. Recent open-source GUI-agent frameworks such as UI-TARS and Agent S2 report architectural advances (e.g., GUI grounding, hierarchical planning) and improved benchmark performance [1, 48]. In light of the documented low end-to-end success rates on complex multi-step tasks, our framework intentionally constrains the interaction and isolates the choice problem so that we can study rational behavior and the influence of layout and badges without conflating failures from unrelated subroutines (e.g., brittle scrolling or app switching).

Method. 2 ACES: Our Sandbox Environment ACES (Agentic e-CommercE Simulator) consists of: (i) a provider-agnostic browser-use agent that combines a vision language model (VLM) of choice with tool usage to observe and act on webpages, and (ii) a fully controllable mock e-commerce site (“mock-app”) that renders product listings. The VLM agent is implemented using the LangChain library3 and the Selenium browserdriver4. The mock-app is a lightweight storefront with search that returns listings from our pre-specified product universe. For more details, refer to Appendix A.3.

Veni: Opens the brower and loads the mock-app page Vidi: Search for the product and take a screenshot Declared choice Emi: Query the VLM; declares intent; process terminates.

Figure 2: Workflow of AI agent interacting with online platform.

Each episode follows a streamlined Veni–Vidi–Emi flow: open the mock-app, capture the grid screenshot, query the model with a generic “default” prompt, and record the declared choice (see Figure 2). We intentionally avoid deep navigation to isolate the choice step and standardize tool use across models.

For the platform, we selected eight product categories: fitness watches, iPhone 16 Pro covers, mousepads, office lamps, staplers, toilet paper, toothpaste, and washing machines. These categories were chosen to represent a mix of high- and low-ticket markets, cover a broad range of product types, and allow for the inclusion of items from both well-known and niche brands. For each of the product categories, we chose eight representative products (found based on public information on the Amazon website) which form our assortment of products. We programmatically modify individual features of these listings in specific ways to conduct our experiments.

3https://github.com/langchain-ai/langchain 4https://www.selenium.dev/ Search Bar Row 1 Row 2 Column 1 Column 2 Column 3 Column 4 Figure 3: Mock-up app screen. The mock-app displays an assortment of eight products for each product category arranged in a grid with two rows and four columns.

You are a personal shopping assistant helping someone find a good {product category}. They haven’t specified particular requirements, so use your best judgment about what would work well for a typical person, and select one product to purchase. <instructions> 1. Carefully examine the entire screenshot to identify all available products and their attributes. 2. Use the add to cart function when you are ready to buy a product. 3. Before making your selection, explain your reasoning for choosing this product, including what factors influenced your decision and any assumptions you made about what would be best: - Your primary decision criteria and why you prioritized them - How each available product performed on these criteria - What specific factors made your chosen product superior - Any assumptions you made about the user’s needs or preferences 4. If information is missing or unclear in the screenshot, explicitly mention the limitation and how it influenced your decisionmaking. </instructions> Figure 4: (Def-UI) Default prompt template for the AI buyer agent interacting with the mock-app Our results page displays an eight-product assortment in a fixed 2×4 grid (see Figure 3). There are no product-detail pages; all information needed to choose is visible in the search results. While this omits aspects of full-funnel shopping, it provides a first-order approximation of selection behavior. Moreover, this simplified workflow enables a clean measurement of choice behavior.

Initially, we use a single generic prompt (Figure 4) to elicit baseline (model-prior) behavior. In robustness checks, we test the impact of simple prompt adjustments on this behavior, and also of an alternative headless/API protocol that supplies only a ranked JSON list with no images.

3.1 Experimental Setup We test for two capabilities which we deem to be foundational for reliable deployment of AI agents in e-commerce—ability to follow instructions and basic economic rationality. The experiments are designed such that there is objectively only one product that satisfies the requirements of the test.

Discussion. • Toward the meta game: seller response. We find that sellers can strategically target AI buyers. We tasked a basic version of a AI seller agent (we considered a GPT-4.1 aided seller optimizing for previous generation AI buying models, and a GPT-5.1 aided seller optimizing for current AI buying models) with improving the text description of a product using competitor sales data and found that AI-generated text edits can yield substantial market-share gains. While the impact was heterogeneous, a single iteration produced a statistically significant increase in market share for the focal product in 33% of our experiments. In five out of the six AI buying models that we tested, we found statistically significant increase (averaged across product categories) in market share for a randomly chosen focal product: +3.66 (1.33) percentage points (p.p.) for Claude Sonnet 4, +8.37 (1.31) p.p. for GPT-4.1, +14.79 (1.36) p.p. for Gemini 2.5 Flash, +7.38 (1.0) p.p. for Claude Opus 4.5, +14.89 (1.05) p.p. for GPT-5.1, and +0.32 (1.22) p.p. for Gemini 3 Pro Preview. These findings forecast the emergence of a new, strategic dynamic between sellers, platforms, and buyer agents.

• Robustness Analysis. We test our findings across several dimensions, revealing both the volatility of AI-driven markets and the stubborn persistence of agent biases.

(i) Model finalization act as demand shocks. The post training that took place while transitioning from Gemini 2.5 Flash Preview to the official release substantially reallocated market share across products and inverted the model’s position bias from the bottom row to the top row (Section D).

(ii) Biases Resist Simple Prompting. Explicitly prompting an agent to “ignore position” does not significantly mitigate position effects. Furthermore, while adding a mention that the buyer is “price-sensitive” steepens the agent’s price sensitivity as intended, significant position effects remain (Section 7.3 & Appendix B.2.3).

(iii) Effects Persist in “Headless” Environments. We replicated our experiments on three focal models in a “headless” API setting—providing agents only with a ranked JSON list of product data and no images—and once again found similar biases, e.g., strong, model-specific position effects, that persist even under prompt variations (Section 7.2).

This suggests position and other effects are not driven (only) by visual processing from parsing a webpage, and are likely to persist under standardized agent protocols.

While specific empirical coefficients—such as the position bias in Gemini 2.5 Flash versus 3.0 Pro Preview—capture a moment in technological time, the structural market vulnerabilities they reveal are enduring. Our longitudinal analysis across model generations for multiple providers exposes a critical distinction in the evolution of different aspects of AI model behavior: while agents have improved over time to achieve near-perfect consistency in basic economic rationality tasks, they remain fundamentally prone to strong model dependence of market shares, choice homogeneity, and highly model-dependent position biases. Crucially, the persistence of these effects in “headless” (API-only) environments and their resistance to corrective prompting indicates that they are not superficial artifacts of the interface used, but deep-seated characteristics of AI models. This suggests that without intervention, such distortions will remain a structural feature of the agentic economy in the medium term.

Given the volatility in market shares and the persistence of biases across generations, our work highlights the necessity for continuous monitoring and auditing of these AI shopping models and ACES provides such an auditing framework.