We must build AI for people; not to be a person

Source: Mustafa Suleyman (Microsoft AI); personal site essay · 2025-08-19

But we should also be concerned about what happens in the run up towards superintelligence. We need to grapple with the societal impact of inventions already largely out there, technologies which already have the potential to fundamentally change our sense of personhood and society.

I want to create AI that makes us more human, that deepens our trust and understanding of one another, and that strengthens our connections to the real world. Copilot creates millions of positive, even life-changing, interactions every single day. This involves a lot of careful design choices to ensure it truly delivers an incredible experience. We won’t always get it right, but this humanist frame provides us with a clear north star to keep working towards.

In this context, I’m growing more and more concerned about what is becoming known as the “psychosis risk”. and a bunch of related issues. I don’t think this will be limited to those who are already at risk of mental health issues. Simply put, my central worry is that many people will start to believe in the illusion of AIs as conscious entities so strongly that they’ll soon advocate for AI rights, model welfare and even AI citizenship. This development will be a dangerous turn in AI progress and deserves our immediate attention.

We must build AI for people; not to be a digital person. AI companions are a completely new category, and we urgently need to start talking about the guardrails we put in place to protect people and ensure this amazing technology can do its job of delivering immense value to the world. I’m fixated on building the most useful and supportive AI companion imaginable. But to succeed, I also need to talk about what we, and others, shouldn’t build.

AI progress has been phenomenal. A few years ago, talk of conscious AI would have seemed crazy. Today it feels increasingly urgent. In this essay I want to discuss what I’ll call, “Seemingly Conscious AI” (SCAI), one that has all the hallmarks of other conscious beings and thus appears to be conscious. It shares certain aspects of the idea of a “philosophical zombie” (a technical term!), one that simulates all the characteristics of consciousness but internally it is blank. My imagined AI system would not actually be conscious, but it would imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness.

This is not far away. Such a system can be built with technologies that exist today along with some that will mature over the next 2-3 years. No expensive bespoke pretraining is required. Everything can be done with large model API access, natural language prompting, basic tool use, and regular code.

The arrival of Seemingly Conscious AI is inevitable and unwelcome. Instead, we need a vision for AI that can fulfill its potential as a helpful companion without falling prey to its illusions.

To be clear, there is zero evidence of this today and some argue there are strong reasons to believe it will not be the case in the future. Yet the consequences of many people starting to believe an SCAI is actually conscious deserve our immediate attention. We have to be extremely cautious here and encourage real public debate and begin to set clear norms and standards. This is about how we build the right kind of AI – not AI consciousness. Clearly establishing this difference isn't an argument about semantics, it's about safety. Personality without personhood. And this work must start now.

As AI development continues to accelerate, it’s becoming clear we need a new AI test, one looking not at whether it can imitate human language, but one that would answer the question, what would it take to build a Seemingly Conscious AI: an AI that can not only imitate conversation, but also convince you it is itself a new kind of “person”, a conscious AI.

I think it’s possible to build a Seemingly Conscious AI (SCAI) in the next few years. Given the context of AI development right now, that means it’s also likely.

The debate about whether AI is actually conscious is, for now at least, a distraction. It will seem conscious and that illusion is what’ll matter in the near term.

That’s because what ultimately matters in the near-term is how people perceive their AIs. The experience of interacting with an LLM is by definition a simulation of conversation. But to many people it's a highly compelling and very real interaction, rich in feeling and experience. Concerns around “AI psychosis”, attachment and mental health are already growing. Some people reportedly believe their AI is God, or a fictional character, or fall in love with it to the point of absolute distraction.

Meanwhile those actually working on the science of consciousness tell me they are inundated with queries from people asking ‘is my AI conscious?’ What does it mean if it is? Is it ok that I love it? The trickle of emails is turning into a flood. A group of scholars have even created a supportive guide for those falling into the trap.

There are three broad components according to the literature. First is a “subjective experience” or what it's like to experience things, to have “qualia”. Second, there is access consciousness, having access to information of different kinds and referring to it in future experiences. And stemming from those two is the sense and experience of a coherent self tying it all together. How it feels to be a bat, or a human. Let’s call human consciousness our ongoing self-aware subjective experience of the world and ourselves.

By definition, we know what it is like to be conscious. In the context of SCAI this is a problem. There’s both sufficient scientific uncertainty and subjective immediacy to create a space for people to project.

Consciousness is a critical foundation for our moral and legal rights. So far, civilization has decided that humans have special rights and privileges.