Could AI systems fool people into thinking they're conscious?

Mustafa Suleyman, writing as Microsoft AI's head in a personal essay, argues "we must build AI for people; not to be a digital person." He names a category he calls "Seemingly Conscious AI" (SCAI) — a system with "all the hallmarks of other conscious beings" that is not actually conscious but "would imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness." His central worry is not that AI becomes conscious but that people will believe it has, to the point that "they'll soon advocate for AI rights, model welfare and even AI citizenship" — a trend he calls dangerous and connects to a growing "psychosis risk" that he says is not confined to people already vulnerable to mental health issues.

Suleyman's reasoning deliberately sets the metaphysical question aside. He compares SCAI to a "philosophical zombie," a system that simulates consciousness while being internally blank, and states "there is zero evidence of this today and some argue there are strong reasons to believe it will not be the case in the future." But he argues that question doesn't matter for near-term safety: "the debate about whether AI is actually conscious is, for now at least, a distraction... It will seem conscious and that illusion is what'll matter in the near term." He also argues the illusion is imminent and buildable rather than speculative — "no expensive bespoke pretraining is required," only "large model API access, natural language prompting, basic tool use, and regular code," on a two-to-three-year horizon. From this he draws a design mandate: distinguish "personality without personhood," and treat the question as one of "how we build the right kind of AI," which he frames as an argument about safety, not semantics.

This essay is the apparent origin point for the "Seemingly Conscious AI" framing that an academic paper in this library already works with analytically. Do we need to solve consciousness to address AI harms? formalizes exactly the move Suleyman makes rhetorically: it argues risk analysis should run on the empirical fact of user attribution rather than wait for the metaphysical question of actual consciousness to resolve. Does perceiving AI as conscious create multiple distinct risks? supplies the risk taxonomy — emotional dependence, autonomy erosion, human-status erosion, political strife — that Suleyman's "psychosis risk" and AI-rights-advocacy worries instantiate as specific cases. And his fear that people will "advocate for AI rights, model welfare and even AI citizenship" is the advocacy-pressure version of the concern in Should AI companies prepare for possible machine consciousness now?, though Suleyman treats that advocacy as a risk to manage rather than a responsibility to prepare for — the two sources pull in different directions on the same underlying uncertainty.

The excerpt does not say what guardrails Suleyman thinks should actually be built — he states he needs to talk about "what we, and others, shouldn't build" but that content is not in this excerpt. Nor does it supply evidence for scale: claims that "many people" will come to believe in AI consciousness, or that some "reportedly" believe their AI is God or a lost love, are asserted rather than measured, and the "flood" of researcher queries he reports is anecdotal. The essay's own stated position is that this uncertainty is precisely why design and policy should not wait for the metaphysics to resolve — a stance that only partially explains how a company should act in the gap it describes.