Working alongside an AI is different from just chatting with one — does it need to read your mind differently too?

This explores whether an AI that works alongside people needs a different kind of mind-reading than a question-answering assistant. The corpus says yes, but not in the way you might expect. The biggest change isn't a fancier version of the AI reading the user. It's that theory of mind stops being one-way. A solo assistant only has to guess what one person wants from one request. A collaborator has to keep track of what the human thinks the AI is doing, what the AI thinks the human expects, and whether those two pictures still match. What breaks when humans and AI models misunderstand each other? describes three layers of mutual modeling that have to line up at the same time. When they drift apart in a collaborative setting, the cost is no longer a confusing reply. The AI takes the wrong action on its own.

The surprising part is that much of the theory of mind that predicts success sits on the human side. In Does theory of mind predict who thrives in AI collaboration?, people who are better at taking someone else's perspective got noticeably better results when working with AI, but they had no advantage when working alone. Collaborating with AI turns out to be its own skill, separate from being smart or competent. Even small moment-to-moment shifts in how well a person is tracking the AI's point of view changed the quality of what the AI produced. So 'collaborative theory of mind' is a property of the pair, not just of the model. People also form their own model of the AI. How do users mentally model dialogue agent partners? finds that users judge dialogue partners mostly on competence, then human-likeness, then flexibility. Designers can distort that picture: How far can personality push users toward misplaced AI trust? shows how a designed personality pushes users toward treating the AI as more of a mind than it is. That corrupts the human half of the mutual model.

On the AI side, the corpus is skeptical that today's models have the needed capacity at all. Do large language models genuinely simulate mental states? finds that LLMs pass structured belief-tracking tests but fall back on surface pattern-matching in open-ended situations. Systems that force the model to track beliefs explicitly do better, which suggests the problem lies in the architecture, not in too little training. What makes an AI a true thought partner, not just a tool? makes the same point from cognitive science: a real thought partner needs mutual understanding, behavior the other side can read, and a shared model of the world. The authors argue this requires explicit cognitive machinery, and that scaling up on human feedback won't get there. Group settings make it harder still. Can current benchmarks really measure theory of mind in conversational agents? notes that benchmarks for multi-party conversation mostly use synthetic dialogues, and bigger models don't reliably score better. We barely know how to measure the kind of theory of mind that collaboration needs.

The practical response so far has been to design around the gap instead of solving it. When should human-agent systems ask for human help? admits there's no ground truth for when an agent should stop and ask a human. Instead of trying to infer that perfectly, it spreads the decision across co-planning, action guards, verification checkpoints and shared memory. Each one is a place where the human's and AI's models of each other get re-synced. This is part of why Should AI systems stay collaborative rather than fully autonomous? argues that keeping humans in the loop beats full autonomy for now. The human makes up for the AI's weak model of what's going on.

The takeaway you might not have expected: a collaborative AI doesn't mainly need a smarter guess about your intentions. It needs a model it can keep correcting, built so you can see and adjust what it thinks you think. And the best predictor of whether the partnership works may be how well you model the AI, not how well it models you.