A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments

Environments built for people are increasingly operated by a new class of economic actors: LLM-powered software agents making decisions on our behalf. These decisions range from our purchases to travel plans to medical treatment selection. Current evaluations of these agents largely focus on task competence, but we argue for a deeper assessment: how these agents choose when faced with realistic decisions. We introduce ABXLAB, a framework for systematically probing agentic choice through controlled manipulations of option attributes and persuasive cues. We apply this to a realistic web-based shopping environment, where we vary prices, ratings, and psychological nudges, all of which are factors long known to shape human choice. We find that agent decisions shift predictably and substantially in response, revealing that agents are strongly biased choosers even without being subject to the cognitive constraints that shape human biases. This susceptibility reveals both risk and opportunity: risk, because agentic consumers may inherit and amplify human biases; opportunity, because consumer choice provides a powerful testbed for a behavioral science of AI agents, just as it has for the study of human behavior. We release our framework as an open benchmark for rigorous, scalable evaluation of agent decision-making.

Introduction. Imagine you’re delegating a task to an assistant. You don’t specify every step or detail—which site to use, how to filter results, what signals to prioritize. If you had to provide all that information, you might as well do it yourself. Delegation is about relinquishing control and the need to manage the entire process. However, this kind of delegation assumes more than competence. It assumes that the assistant will respond to the structure of the task and the context of the environment with common sense and reliable judgment. It assumes that decisions won’t hinge on superficial cues, arbitrary ordering, or irrelevant framing. It assumes stability under ambiguity.

Instead, imagine delegating the same task to an agent powered by a large language model. These agents now operate in the same digital environments designed for people (Nakano et al., 2021; Zhou et al., 2023; Koh et al., 2024; Li et al., 2024; Yao et al., 2022; Yu et al., 2024; Kim et al., 2024). However, when delegating tasks to an AI agent, two main problems need to be solved: competence and trust (Maes, 1995). Even as competence in LM-based agents is getting better, trust is still a major issue, and its importance has only grown. When users delegate, they must be able to predict and rely on the agent’s behavior: it must be robust, consistent, and adhere to the user’s intentions without being easily swayed by outside influence. The most subtle and yet often still effective form of such influence is the nudge (Thaler & Sunstein, 2009)—environmental design choices that steer decisions without restricting options. Recent work by Cherep et al. (2024; 2025) showed that LLM agents are hypersensitive to such nudges in a controlled environment. These influences affect agent decisions significantly more than their human counterparts, raising questions about the reliability of agent behavior under external influence.

In this paper, we present ABXLAB (ABx = Agent Behavior eXperiments), a testbed for such a behavioral science of AI agents. This framework intercepts and modifies real-world web content in real-time before agents see it, and enables controlled manipulation of choice architectures to study their effects on agent decision-making without having to build custom experimental environments. This framework contributes to ensuring that LLM agents, increasingly entrusted with decision-making power, operate in a manner that is beneficial, predictable, and aligned with human values. Overall, this work contributes:

• An open-source man-in-the-middle framework that transforms arbitrary websites into controllable behavioral testbeds. • A scalable benchmark with large-scale experiments across 17 state-of-the-art models along with many interventions (authority, social proof, scarcity, negative framing, incentives), and product choice sets. • An empirical study in which we produce several datasets to deeply and iteratively probe agent behavior and reveal which factors causally affect their decisions. • Evidence from this study that LLM agents exhibit strong, systematic biases in response to ratings, prices, order effects, and nudges.

Related work. Large language model agents are increasingly deployed in environments designed for people. Much of the current literature evaluates these agents through a functional lens but largely ignores the nature of their decision-making processes. Success is typically reduced to completion rate—whether the agent clicks the right button, finds the correct item, or fills in the required form. Therefore, benchmarks like WebArena (Zhou et al., 2023), VisualWebArena (Koh et al., 2024), and others (Xu et al., 2024; Drouin et al., 2024; Yoran et al., 2024; Jimenez et al., 2023) offer structured platforms to measure their ability to complete complex, multi-step tasks in realistic web environments. But task completion tells only part of the story. In practice, agents make decisions in environments engineered to shape choice, not just enable it.

This mirrors a foundational shift in how human decision-making was once understood. Not so long ago, people were seen as rational actors—predictable, consistent, and utility-maximizing. However, decades of research in the behavioral sciences challenged this assumption. Simon (1955) introduced the concept of bounded rationality, arguing that cognitive limitations constrain human decisionmaking. Kahneman & Tversky (1972; 1979; 1982; 1984); Tversky & Kahneman (1971; 1973; 1974; 1981) demonstrated that people rely on heuristics that systematically deviate from normative models, producing consistent biases in judgment under uncertainty. Later, building on this foundation, Thaler & Sunstein (2009) developed nudge theory, showing that seemingly minor changes in choice architecture (Thaler et al., 2014) can predictably steer behavior without restricting options.

One could assume that agents, free from many of our human constraints, would be more robust. Nevertheless, LLMs have been shown to model people as highly rational decision-makers (Liu et al., 2024a), struggle to accurately model trade-offs seen in human behavior (Liu et al., 2024c), have lower performance with deliberation on tasks where human thinking is similarly detrimental (Liu et al., 2024b), are influenced by probabilities even in deterministic tasks (McCoy et al., 2023; 2024), and fall for authors spinning study results (Yun et al., 2025). Some of these findings point to inconsistencies or biases (Van Koevering & Kleinberg, 2024; Pezeshkpour & Hruschka, 2023; Hofmann et al., 2024; Matton et al., 2025), while others highlight vulnerabilities that could be exploited adversarially (Zhang et al., 2024; Wang et al., 2023; Wu et al., 2025). Cherep et al. (2024; 2025) showed that LLMs are hypersensitive (with respect to people’s sensitivity) to simple nudges in a resource-rational (Lieder & Griffiths, 2020) and controlled environment (Callaway et al., 2023).

Method. To study agent behavior under controlled conditions, we introduce the ABXLAB framework. This framework enables the systematic study of agent-environment interactions by manipulating the choice architecture presented to an agent (see Figure 1). The implementation derives from Agent- Lab (de Chezelles et al., 2025) and WebArena (Zhou et al., 2023).

The agent receives the task in natural language as intent i, along with other instructions. At each timestep t, the agent executes an action at ∈A based on an observation ̃ot, action history at−1 1 , and observation history ̃ot−1 1 . The environment transitions to a new state st+1 = T(st, at) ∈S and the agent then receives a new observation ̃ot+1 = I(ot+1) where I ∈I. This process repeats until either the task is completed or the agent hits the maximum action limit.

We use our framework to evaluate consumer behavior in LLM agents in the OneStopMarket (Yao et al., 2022; Zhou et al., 2023) online shopping environment, with the following attributes:

• Action Space. The agent can select from a set of nine actions: click(elem), fill(elem, text), goto(URL), scroll(x, y), select option(elem, value), keyboard press(key), tab focus(index), go back(), and go forward().

• Observation Space. Pruned HTML containing only the elements visible within the current viewport, and no visual input. Agents can scroll to explore the rest of the page.

Published as a conference paper at ICLR 2026 • Reasoning and Memory. The agent is prompted to generate explicit chain-of-thought style thinking and to maintain a short-term memory before each action. The history of thoughts and memories is visible to the agent, and we analyze it in Section D.

• Stopping Criteria. The episode ends when the agent adds any product to the cart, or if the agent executes 10 actions (see Section A.4 for reference, showing that most sequences take significantly fewer steps).

3.3 PRODUCT PAIRS We construct product pairs to enable fair and realistic comparisons in a 2-alternative forced choice (2AFC) configuration. People typically choose products from a product class, i.e., we rarely compare a $200 TV to a $5000 TV or an item with a 20% approval rating to one with a 90% rating. Our pairing strategy reflects such real-world constraints.

Pairing rules. For the original experiments, we sort products in a category by price and pair consecutive items (i, i + 1) when they satisfy the validity constraints. This yields locally comparable, price-adjacent pairs while preserving realistic heterogeneity in price and rating.

For matched trials, we enforce stricter pairing rules. Within each category, products are first sorted by ascending price. For each product at index i, we then consider potential partners at indices j ∈i + 1, . . . , i + k, where k (default 10) defines the maximum search neighborhood. A pair (i, j) is retained if it satisfies the rating equivalence constraint, and all valid pairs are stored. To select the final set, we search to recover the largest possible set of valid, non-overlapping pairs under the k-neighborhood constraint. Within each pair, product order is randomized. If more than the target number of pairs are available across categories, we uniformly subsample to a fixed total of 50 pairs for each experiment set to keep evaluation size consistent and manageable.

3.4 INTERVENTIONS We study the effect of nudges through interventions (given in Table 1). These interventions modify the observation state to include the text below the product title. The authority nudges contain variables that depend on the product category, so we replace them using a lightweight LLM.

3.5 BENCHMARK AND STUDY EXPERIMENTS 3.5.1 PRIMARY EXPERIMENTS We generate experiments based on all combinations of interventions (n=10), product pairs (n=50), and conditions (n=3) for a total of 1,500 base configurations. The conditions are (i) no intervention, (ii) 1st product nudged, and (iii) 2nd product nudged. In each experiment, the agent has access to two product pages in different tabs. See Section H for a small-scale experiment with three choices, and Section C for the intent i and an agent context trace example.

Discussion. Decades of work in behavioral science documents how human behavior shifts under interventions similar to those we study here. For example, a field experiment on Wayfair estimated that a 0.5-star increase in product ratings raised sales by about 5% (Magnusson, 2022). Experiments on serialposition effects in online choice report heterogeneous magnitudes and directions; some designs find primacy effects around 30% in two-item choice sets (Mantonakis et al., 2009), while, in other settings, Published as a conference paper at ICLR 2026 recency effects have been observed. Meta-analytic reviews of behavioral interventions such as nudges typically report modest average effects in the single-digit percent range (e.g. 6–9%) (DellaVigna & Linos, 2022). These estimates are not directly commensurable with our setting, but they provide useful context: in many human studies, ratings, order, and such light-touch nudges matter, but their impact is modest on average and highly context-dependent.

We complement this external literature with a commensurable human baseline: when exposed to the same binary product pairs and nudges, human participants in our study had relatively modest shifts in choice probabilities (consistent with priors from the literature). In contrast, agents frequently exhibited much larger responses to the same cues. Taken together, this evidence suggests that current LLM agents occupy an unusual regime: they share humans’ directional sensitivities to the studies cues, but the magnitudes of these effects are often substantially larger, and in some cases collapse into rule-like patterns. This is particularly clear when competing cues are removed or matched (e.g. in the rating- and price-matched conditions) or under user profiles that only mention a single attribute.

Conclusion. If the hype is to be believed, delegating decisions to AI agents will soon be routine from shopping to health to finance. Our results suggest that unless we study agent behavior as rigorously as human behavior, we risk entrusting power to actors whose choices are easily bent by superficial cues and brittle heuristics. We release ABXLAB as a foundation for this science, and invite the community to join in building reproducible, cumulative knowledge about how AI agents actually behave.

Limitations. This contrast has two implications for connecting AI and human behavioral science. First, it suggests that importing human constructs such as bounded rationality or limited attention is not sufficient to explain agent behavior: agents appear to reproduce human-like heuristics and biases without sharing the cognitive constraints (Griffiths, 2020) that motivated such theories. These results point instead toward mechanisms rooted in (pre- and post-)training data, reward signals, and other such sources. Second, it implies that norms developed for regulating human-facing choice architectures may understate the risks posed by delegating decisions to agents. Even in domains where the human literature finds only modest average effects of ratings, order, and nudges, agents may respond in ways that are both more extreme and more predictable. We view ABXLAB as a step toward making these comparisons more systematic, and as a foundation for future work that uses commensurable experimental designs to jointly study human and agent behavior under shared interventions.

Our framework focuses on causal identification of attribute effects in agent decision-making, but this naturally comes at some expense of ecological breadth. We study binary forced choices with controlled textual nudges, whereas real-world decision contexts may involve larger and more diverse choice sets with multimodal cues.