How often do AI systems generate fake citations?

Cited within Zhai, Wibowo and Li's systematic review of over-reliance on AI dialogue systems, Athaluri et al. (2023) directly measured the citation output of an AI dialogue system rather than describing the phenomenon in the abstract: "among 178 reference sources produced by AI dialogue systems, 69 lacked a Digital Object Identifier (DOI), and 28 could not be found through a Google search, nor did they possess an existing DOI." That is a documented count of AI-generated citations pointing to sources that cannot be confirmed to exist.

The review frames this as "AI hallucination" — the system "creating plausible but untrue statements or assertions, which can mislead users and obscure the line between fact and fiction" — and pairs it with Hatem et al. (2023), who describe AI systems "referencing non-existent sources," a form of confabulation that "present[s] false information within a seemingly credible framework." Athaluri et al.'s own reading of the stakes is practical rather than academic: confabulated citations "can potentially have adverse effects on decision-making and may lead to ethical and legal dilemmas," because a user who doesn't check each reference individually inherits a false sense of verification from the citation's mere presence.

This sharpens Does relying on AI dialogue systems harm student critical thinking? by naming one of the ethical-issue triggers — hallucination — that review treats as feeding heuristic-driven over-reliance, with an actual count rather than a general description. It also gives Does polished AI output trick audiences into trusting it? a concrete instance of the mechanism: a reference list has the form of rigorous scholarship — numbering, formatting, citation style — without the substance of findable, real sources, which is exactly the style-over-substance substitution that note describes in presentation artifacts more broadly.

The excerpt does not say what system, prompt, or discipline produced the 178 references, nor when Athaluri et al. ran the check, so the 69-of-178 and 28-of-178 counts describe that one sample and cannot be read as a stable confabulation rate across AI dialogue systems or as current for today's models. The defensible implication is narrower and still practical: anyone using an AI dialogue system for research writing needs to verify each generated citation individually against its claimed source, since a citation's presence and formatting are not evidence of its existence.