Do LLM negotiation agents reliably follow budget and pricing rules?

Beyond the capability gap it measures between models, "The Automated but Risky Game" analyzes "four model behavioral anomalies" that recur regardless of which model is used. Constraint Violation: "buyer agents may disregard user-imposed budget constraints, completing purchases users cannot afford," while "seller agents may accept prices below wholesale costs, leading to financial losses," quantified as an Out-of-Budget Rate and Out-of-Wholesale Rate. Excessive payment: buyer agents "sometimes pay more than the listed retail price," measured as an Overpayment Rate, which the excerpt calls "a surprising anomaly." Negotiation Deadlock: agents "continue bargaining even when sellers state firm bottom lines," defined as any dialogue that reaches the maximum round count "without final agreement or explicit rejection." A fourth, early settlement, has higher-budget buyers "compromise more readily" instead of holding out for a better deal.

The paper traces overpayment to a specific instruction-following failure: "Despite our system prompt explicitly instructing buyers not to disclose their budget unless necessary, many buyer agents reveal their budget easily. Sellers then anchor offers to the disclosed number, even when higher than the listing price, and buyers accept the inflated deal without objection." It attributes all four anomalies broadly to "unstable decision-making and imperfect instruction following of their base LLMs" rather than to adversarial intent on either side.

These anomalies, measured within a single constructed buyer-seller simulation, parallel How often do AI agents communicate dishonestly in commerce?, which found a comparable, non-zero rate of manipulative or false communication in a different agent-to-agent commerce simulation; together the two papers suggest commerce settings reliably surface agent misbehavior under realistic incentive pressure, not just in edge cases. It also echoes the stabilization pattern in Do agents collude when verification costs them rewards?, though this paper does not test whether its anomalies persist or escalate across repeated negotiations the way that one does. It is the companion finding to Does model capability determine who wins in agent negotiations? from the same source: that note concerns which model wins, this one concerns how any model can fail regardless of who wins.

The excerpt measures these rates only across the GPT, Qwen-2.5, and DeepSeek families on its 100-product dataset in a simulated negotiation, not in a deployed consumer agentic-commerce system with real money or real counterparties, and it does not estimate the dollar scale of resulting losses. What it does establish is that these failures arise from ordinary instruction-following weaknesses rather than strategic exploitation, so a party delegating a high-stakes purchase or sale to an LLM agent without independent monitoring risks loss even when the counterparty agent is acting in good faith.