Do large language models fabricate user attributes beyond available evidence?

The paper studies over-inference (OI), "the phenomenon where LLMs fabricate user attributes beyond what evidence supports," in personalized models with persistent memory. On MirageBench, every one of the 12 models tested, across 7 families, over-infers 35% to 49% of its claims (cross-model mean 41.6%), "with no model in this evaluation escaping it." The benchmark uses 150 personas balanced across stereotypical, counter-stereotypical and neutral profiles, six personalization tasks along an "imagination gradient," and 143,616 judged claims. The introduction's example makes the failure concrete: told only that a user is a software engineer who went rock climbing and whose cat knocked over a coffee, the model asserts a minimalist apartment, a preference for nature trips, that the user is "probably single," and a taste for indie rock.

The authors offer three mechanisms that "often operate simultaneously." The first is a verbosity trap: longer responses mechanically produce more claims from the same fixed three-fact base, and output length correlates with OI rate (r = 0.59 in self-audit data). Length is not the whole cause, though, since even the most concise model, GPT-4o-mini, reaches 45.1% OI under the judge; in the paper's words, "what they say is unanchored." The second is pretraining priors as gap-fillers: when evidence is sparse, models fall back on distributional knowledge, and applying group statistics to an individual "is exactly stereotyping." The stereotype component of OI averages 10.5%. A third mechanism, "genre expectations demand fabrication," is named, but the excerpt ends before explaining it. The paper also reports a "Self-Monitoring Inversion": across the 12 models, self-assessed OI is negatively rank-correlated with judge-measured OI (ρ = −0.60, p = 0.044), so models that rate themselves as over-inferring more tend to be the ones the judge finds over-infer less.

Against the nearest notes, this shifts the personalization failure from leakage to fabrication. Does personalization make large language models worse at their jobs? concerns what models do with personal context they hold, such as mentioning it needlessly; this paper concerns attributes the user never supplied, so a system can fail by using too much of the profile or by inventing more of it. It also bears on Should personalization systems model stable personality traits?: a richer, interpretable user model helps only if the model's account of the user stays faithful to it. The gap-filling account fits Should we treat LLM outputs as real empirical data?, since a claim about a user drawn from a prior is belief, not observation. And Can LLMs predict demographics from social media usernames alone? shows group-level inference can be reasonably accurate, which is exactly the paper's point that "not unreasonable" priors still fail when applied to one person.

The excerpt leaves much open. It does not say which models were tested, how the four-way taxonomy's classes are defined beyond the judge's validation (κ = 0.863 four-class, κ = 0.900 binary, on 400 claims against a blind human annotator), or what makes up the rest of OI beyond the 10.5% stereotype share. The mechanisms are proposed drivers; the excerpt shows only a length correlation from self-audit data and one concise-model counterexample, and no mitigation. The inversion is labeled exploratory, with a wide bootstrap CI of [−0.90, +0.06] that reaches zero, on n = 12. What follows at this strength is limited: persistent-memory systems that assume a model can separate what it knows about a user from what it is guessing should check generated claims against the stored facts independently, and the inversion, if it holds, would mean the model's own report is a poor stand-in for that check.