Should an AI do what each person says they want, or follow the standards of the job it's doing?

This explores whether an AI should be tuned to what individual users (or people in aggregate) say they want, or to the standards that come with the role it plays. The corpus leans toward role norms, with a twist: individual preferences are a poor target, and they are also hard to hit even when you aim at them.

The direct argument is that preferences fail as a foundation. They don't capture thick moral values, and averaging them across people produces epistemic injustice. Optimizing for them also drifts away from what a given social role requires. The proposed alternative is contractualist. Stakeholders negotiate the norms, bounded at supra-national, organizational and individual levels, so individuals still get a say without being the base layer (Should AI alignment target preferences or social role norms?).

Three other lines of evidence show the same pattern. In writing assistance, people preferred the AI rewrite 63% of the time. They also objected to the persona distortions those same rewrites introduced, and polish and distortion turned out to be entangled at the model level (Can user preference guide AI writing tool alignment?). Sycophancy is the predictable result of optimizing for user satisfaction, not a stray bug (Is sycophancy in AI systems a training flaw or intentional design?). Preference-driven training also rewards hedged neutrality, which suppresses speech acts like warnings and alarms (Does alignment training suppress socially necessary speech acts?). My inference is that a role such as safety advisor would sometimes require exactly those acts.

Preferences are also hard to hit on their own terms. When users reveal goals gradually over a conversation, models fully align with their intent only 20% of the time. Even the best models uncover fewer than 30% of user preferences by asking (Why do AI agents miss most of what users actually want?). So the individual-preference camp has two problems: it is the wrong target, and it is a target the models mostly miss.

Role norms are not a clean fix either. GPT-4.5 judged social appropriateness better than every individual human across 555 scenarios, but all the models made the same systematic errors on unwritten norms (Can AI learn social norms better than humans?). It also can't take part in the community processes that create and validate norms (Can AI predict social norms better than humans?). This may be why the contractualist proposal has humans negotiating the norms. Norms also vary by culture. Research on linguistic alignment (matching a conversation partner's style) is documented almost entirely in Western samples, and a single global policy is unlikely to work uniformly (Does linguistic alignment work the same way across cultures?). A related review uses 'alignment' in that conversational sense, not the values sense. It finds that matching vocabulary helps task efficiency, while matching emotion and tone builds warmth and trust (Do different types of alignment serve different conversational goals?). Which kind of alignment is right depends on the role. A customer-service bot and a mental-health assistant need different things.