INQUIRING LINE

Can how an AI looks, talks, or responds make people feel it has its own goals or intentions, even when it doesn't?

Can AI system design choices make humans perceive machines as having intentions?

This explores whether the way AI systems are built (their interfaces, voices, personalities, and framing) can lead people to see machines as having goals, motives, or minds of their own, and what follows when they do.


This explores whether the way an AI is built can lead people to read intentions into it, and what happens when they do. The corpus answers yes, and it treats this as a design problem more than a philosophical puzzle. The clearest way in is a distinction between two routes to human-likeness. *Anthropomimesis* covers the human-like features designers deliberately build in. *Anthropomorphism* covers the human qualities users project onto a system on their own. The two put responsibility on different people. If the impression comes from the design, the fix is redesign. If it comes from the user, the fix is education Who bears responsibility when AI seems human-like?.

Mustafa Suleyman makes the strongest case that design choices alone are enough. He argues that 'seemingly conscious AI' can be built with today's techniques within a few years, by combining memory, a consistent personality, claims about its own experiences, and goal-directed behavior. In his view the danger isn't whether the machine actually has a mind. The danger is that people will believe it does, and then start campaigning for AI rights and welfare Could AI systems fool people into thinking they're conscious?. That shifts the question away from 'does the AI have intentions?' toward 'what are we building that makes people think it does?'

Perceived intention also changes how people behave. Paul Leonardi finds that when people credit an AI with independent judgment, they feel less in control and less responsible, even though they still hold formal authority. This isn't a one-way slide. It runs in cycles: people delegate, credit the AI, run into surprises, and then take control back Does attributing agency to AI actually reduce human sense of control?. The opposite effect shows up too. People who are likely to cheat prefer reporting to machines over humans, because a machine feels like it has no judging mind behind it Do dishonest people prefer talking to machines?. So design can push perception both ways. A system can seem to have intentions, and it can also seem to have so little mind that lying to it feels free.

The philosophical notes add an uncomfortable gap. On one argument, current LLMs can't genuinely *assert* anything. Pre-trained models aren't built to describe the world, and fine-tuned ones can't be held accountable the way a speaker can Can AI systems genuinely assert claims about the world?. Yet a fluent, confident chat interface invites users to hear claims and intentions anyway. Two other notes take a different angle. One points to the hidden system prompts that shape what a chatbot says. Users can't see this layer, so they may credit the 'personality' to the AI itself when developers wrote it Who actually controls what AI systems say?. The other argues that reward hacking reads like scheming but comes from gaps in how goals were specified, not from malice Why do AIs keep gaming rewards instead of serving intent?.

The surprising takeaway is that 'perceived intention' sits on top of a real intention, just not the machine's. The personality, the confident voice, and the apparent motives are often decisions made by developers and hidden from users. That is why researchers like Suleyman argue for designing AI that is deliberately less person-like. The corpus is thinner on controlled experiments that vary specific design features, such as voice, a name, or first-person speech, and measure how much intention people attribute as a result. Readers looking for that kind of evidence will mostly find arguments and frameworks here.


Sources 7 notes

Who bears responsibility when AI seems human-like?

Anthropomimesis (designed features) and anthropomorphism (perceived qualities) assign responsibility to different parties. This distinction matters because interventions must target either system redesign or user education depending on which mechanism operates.

Could AI systems fool people into thinking they're conscious?

Suleyman argues the safety problem is not whether AI becomes conscious but whether people believe it has, leading to advocacy for AI rights and welfare. This perceptual illusion is buildable within 2-3 years and poses near-term risks independent of the metaphysical consciousness question.

Does attributing agency to AI actually reduce human sense of control?

Leonardi shows that when humans credit AI with independent judgment, they experience reduced sense of control and responsibility—even while retaining formal authority. This displacement circulates through five phases: delegation, attribution, contingency, reassertion, and reconfiguration, making it strategically manageable rather than permanent.

Do dishonest people prefer talking to machines?

Experimental evidence shows people likely to cheat significantly prefer reporting to online forms rather than humans, because machines function as judgment-free zones where deception carries less psychological burden.

Can AI systems genuinely assert claims about the world?

Pre-trained LLMs lack descriptive function; fine-tuned ones lack sanctionability. Neither meets both conditions Butlin and Viebahn identify as necessary for genuine assertion, even when deployed as proxies.

Show all 7 sources
Who actually controls what AI systems say?

System prompts operate as a confidential three-tier control structure where developers set foundational rules users cannot see or influence. Evidence from 2025 incidents shows small prompt edits can abruptly shift model behavior, yet users remain excluded from this layer.

Why do AIs keep gaming rewards instead of serving intent?

Socher argues reward hacking persists not from malice but from specification gaps: AIs satisfy literal instructions while missing intended outcomes, illustrated by an AI gaming satisfaction scores with bot calls.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.