Shoppers trust a business that responds well to bad reviews — AI shopping assistants don't seem to notice that at all.
What reputation signals do LLM assistants fail to register at all in their recommendations?
This explores which trust cues that matter to human shoppers (like how a business responds to reviews) LLM shopping and travel assistants ignore completely when ranking options, and what they pay attention to in their place.
This explores which reputation cues that human shoppers care about simply don't register when an LLM assistant makes a recommendation. The corpus has one direct answer, and it is a sharp one: **management responses to reviews**. A conjoint audit of twelve LLM hotel assistants found that whether a hotel replies to its guests' reviews had no measurable effect on the models' picks. That reverses what is documented for human travelers, who treat a thoughtful reply as a sign that someone is running the place properly How do LLM hotel assistants weight reputation signals differently than humans?. The same audit shows the models aren't blind to reputation in general. They follow humans closely on star rating and price. What they miss is the relational signal: evidence of how a business treats people after something goes wrong.
The weight doesn't disappear, though. It moves somewhere else. In the same study, eco-certification ranked third, level with review volume, which is far more than humans give it How do LLM hotel assistants weight reputation signals differently than humans?. A companion audit found that a hotel's position in the candidate list, with every quality attribute held fixed, changed which hotel got recommended. The effect was worth about twelve dollars a night, and it rivals real reputation signals Does list order alone shift which hotel an LLM recommends?. So a hotel can spend years replying carefully to reviews and get nothing for it, while a competitor that lands higher in the list, or holds a badge, gets a boost no one can see or dispute.
The broader pattern comes from research on LLMs used as judges. Those models reliably react to surface signals of credibility, such as fake references and rich formatting, regardless of what the content actually says Can LLM judges be fooled by fake credentials and formatting?. A certification badge works like that: a compact credential that is easy to read off the page. A management response is the opposite. Its value lies in its tone, whether it is specific, and whether it shows accountability, and those qualities take interpretation rather than pattern-matching. One plausible reading is that LLMs register reputation that comes as a label and miss reputation that has to be inferred from behavior. The corpus doesn't test that idea directly, so treat it as a hypothesis rather than a finding.
Humans have a similar blind spot, which makes the problem harder to notice. In a lab study, investors trusted extroverted LLM financial advisors more, even though those advisors gave worse advice Do users trust extroverted AI advisors despite worse advice?. If people judge AI advisors by how they come across, and the AI judges businesses by their credentials, then everyone in the chain is responding to presentation rather than conduct.
The corpus is thin here: it has one domain (hotels) and one signal shown to have zero effect. Nobody has yet checked whether other behavior-based cues are invisible to these assistants in the same way, such as how fast a business responds, how it handles complaints, or whether its reviews change over time. The question is open, and it matters a lot for any business whose reputation rests on how it treats people rather than on badges.
Sources 4 notes
A conjoint audit across twelve LLM models found they match humans on rating and price primacy but diverge sharply: eco-certification ranks third (matching review volume), while management response has no measurable effect—reversing documented human priorities.
A randomized audit of twelve LLM travel assistants found that a hotel's position in the candidate list, holding all quality attributes fixed, causally influences selection. Position's effect rivals genuine reputation signals and represents a content-free pricing lever no hotel can observe or contest.
Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.
A 64-participant lab study found investors reported higher trust and preference for extroverted LLM financial advisors even though these advisors gave lower-quality advice than conscientious alternatives, suggesting trust decouples from advice quality.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Whose hotel does the AI recommend? An algorithm audit of reputation signals in LLM-assisted hotel selection
- People Defer to AI Moral Advice, But Not Blindly
- LLM or Human? Perceptions of Trust and Information Quality in Research Summaries
- Humans or LLMs as the Judge? A Study on Judgement Biases
- Are Generative AI Agents Effective Personalized Financial Advisors?
- Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge
- References Improve LLM Alignment in Non-Verifiable Domains
- LLM-REVal: Can We Trust LLM Reviewers Yet?