Does pooling AI brand mentions mask the true recommendation effect?

Measuring every AI brand mention as if it were a recommendation produces a bigger, more attractive number than the real effect. The paper reports that "pooling every mention gives an attractive but confounded funnel (+2.08/+2.37/+0.52 pp)" for search, own-site, and retail lift — roughly double the +4.3/+2.4/+1.0 pp the paper recovers once it isolates genuine recommendations made to users with no recent engagement with the named brand. The gap between the two numbers is the size of the confound, not noise.

The paper traces the inflation to two distinct sources mixed into the pooled average. First, "many mentions are incidental references to brands the user already uses ('your Netflix download')," and the downstream visits attached to those mentions are just that existing customer's own ordinary behavior, not a response to being told about the brand. Second, and more subtly, a brand's own-site activity is often already rising in the days before the assistant's response — a "brand-specific pre-trend" — so some of what looks like a caused visit was already underway. An event study makes this visible directly: "the named brand's own-site activity is already rising in the days before the response, while same-category brands the response did not name stay flat." The paper's fix is to condition on non-customers (no prior search, site, or retail visit in the lookback window) and to classify each mention's stance (recommend / neutral / caution) with a small language model, which together strip out both confounds and leave the acquisition-like effect: "incidental name-drops then move behavior two to three times less" than genuine recommendations.

This is a methodological point as much as an empirical one, and it generalizes past this specific panel: any measurement that ties an AI brand mention to a later outcome and does not separate existing customers from new exposure, and genuine recommendations from incidental references, is measuring a number inflated by demand that already existed. It is the general caution behind neighboring notes that quantify AI-mediated brand exposure from the other side — How consistent are AI brand recommendation lists across repeated prompts? and Do AI search engines favor well-known brands over others? both count mentions or citations without asking who the user already was when the mention occurred; this paper's decomposition is the check that question needs before a mention-count is read as a causal exposure. It also sits next to Why do consumers delay adopting agentic search?: both papers warn that an aggregate, pooled measurement of AI-mediated commerce can misstate who is actually affected and by how much.

The decomposition is specific to this panel's instruments — a stance classifier, a non-customer definition built from the panel's own clickstream history, a same-category control — and the paper is explicit that even its cleaned estimate cannot fully rule out "a brand-specific intent shock that, in the same session, both prompts a recommendation and drives action," since that would need a randomized or encouragement design to close. What the decomposition does establish is narrower but still load-bearing for anyone citing a mentions → visits number from this kind of study: the pooled, unconditioned version of that number should be treated as an upper bound on the causal effect, not the effect itself.