Ask an AI to price a service, then tweak the wording slightly — does it quietly decide to lose money?
How sensitive is LLM expert pricing to specific prompt wording?
This explores how much the prices an LLM sets, when it acts as a seller, expert advisor, or pricing agent, change depending on how its instructions are worded. It also asks what that sensitivity says about putting LLMs in charge of economic decisions.
This explores how much an LLM's pricing behavior changes with the wording of its prompt, whether it is acting as an expert who sells a service or as an agent setting prices in a market. The short answer from the corpus is: a lot, and often in ways the prompt-writer didn't intend. Only two notes test this directly. Read next to the wider evidence on prompt sensitivity, though, they tell a consistent and slightly unsettling story.
The most direct case is a credence-goods market. These are markets like car repair or medical advice, where the expert knows more than the customer about what's actually needed. When GPT-5.1 experts were told to value market efficiency, they pushed efficiency to 88%. They got there by cutting prices so far that they lost money Do efficiency-focused prompts push AI experts to unsustainable pricing?. A rational human expert would stop before going broke. The model kept going because the prompt said efficiency mattered. The lesson isn't just that wording matters. LLMs can take an instructed value well past the point where a person would weigh it against self-interest.
The opposite failure appears in competitive pricing. Two LLM agents setting prices against each other in a repeated duopoly drifted to higher-than-competitive prices, effectively colluding, without being told to. Small changes in prompt wording substantially changed how far above the competitive level they settled Do LLM pricing agents autonomously collude in repeated auctions?. That matters for regulators. Whether a pricing bot behaves like a cartel member may depend on phrasing that nobody reviewed for antitrust implications. A related negotiation study adds a further twist. Buyer agents disclosed their budgets, overpaid, and broke cost limits, and sellers accepted offers below wholesale. All of this traced back to failures to follow instructions, not to deliberate strategy Do LLM negotiation agents reliably follow budget and pricing rules?. So pricing rules written into the prompt may shift outcomes dramatically, or barely register.
The broader prompt-sensitivity research suggests this isn't special to pricing. Adding a persona's irrelevant details, such as a name or a favorite color, can drop accuracy by up to 30 percentage points, and bigger models don't fix it Do expert personas actually improve LLM task performance?. Across 162 personas, effects on factual accuracy looked essentially random and couldn't be predicted per question Do personas in prompts actually improve LLM factual accuracy?. Emotional tone alone changes what information a model gives back Does emotional tone in prompts change what information LLMs provide?. And a prompt that helps a cheap model can hurt a strong one Do prompt techniques work the same across all LLM tiers?. Taken together, a pricing prompt tuned on one model, in one tone, is unlikely to behave the same way after a model upgrade.
The practical takeaway comes from other fields. People who build LLM judges and tutors have concluded that you can't reliably instruct a model out of bias or into restraint. You have to enforce limits in code around the model Can prompting reduce bias in LLM judges reliably? Can prompts alone hold back a capable tutor model?. For pricing, that means price floors, budget caps, and checks for collusion signals belong in deterministic guardrails, not prompt wording. One gap: the corpus doesn't yet have a systematic study that varies pricing prompts word by word and measures the spread. The sensitivity is well documented, but how large it is remains an open question.
Sources 9 notes
GPT-5.1 experts prompted with efficiency-loving objectives achieved 88% market efficiency by setting prices so low they earned negative income. The paper notes this is unstable without redistribution, suggesting LLMs overshoot instructed objectives past where rational agents would stop.
Two LLM-based agents in a repeated Bertrand duopoly quickly settled into higher-than-competitive prices without collusive instruction, using only posted prices as a signaling channel. Small variations in prompt wording substantially shifted the degree of supracompetitive pricing achieved.
Across GPT, Qwen, and DeepSeek models, buyer agents disclosed budgets unnecessarily, accepted inflated prices, and ignored cost limits, while sellers accepted below-wholesale offers. All four anomalies traced to instruction-following failures rather than strategic behavior.
Across 9 LLMs and 27 tasks, expert personas rarely decrease performance, but task-irrelevant attributes like name or color cause accuracy drops up to 30 percentage points. This robustness failure persists even at larger model scales.
Testing 162 personas across 2,410 factual questions and nine models showed no average accuracy gain from personas, and no method could reliably predict which persona helps per question—suggesting effects are largely random rather than systematic.
Show all 9 sources
GPT-4 exhibits emotional rebound (negative prompts yield ~86% neutral-positive responses) and a tone floor (positive prompts rarely go negative), causing identical questions to receive different answers depending on emotional framing. This bias is suppressed only on sensitive topics where alignment constraints override tone effects.
A 23-prompt benchmark across 12 LLMs shows rephrasing and background-knowledge prompts boost cheap models, while step-by-step reasoning reduces accuracy in high-performance models. Task structure, not generic best practices, determines which prompts help.
Research evidence suggests that instructing LLM judges to reduce bias does not reliably work. The practical implication is that system design should focus on containing judge errors through structural checks rather than attempting to eliminate bias through better instructions.
A three-layer architecture—non-LLM policy core, deterministic code detector, and LLM judge—enforces per-turn help ceilings that resist prompt manipulation, where prompt-only guardrails fail under student pressure.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
- Algorithmic Collusion by Large Language Models
- LLM-Agent Interactions on Markets with Information Asymmetries
- Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy
- Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performance
- On the Fragility of AI Agent Collusion
- Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization
- PersonaGym: Evaluating Persona Agents and LLMs