What makes a process for choosing between values legitimate and fair?
This explores what makes a *procedure* for resolving value conflicts trustworthy — not which values win, but how the choosing gets done so the outcome counts as legitimate and fair.
This reads the question as being about process, not verdict: when values collide, what makes the *way we decide between them* something people can accept as fair? The corpus has a surprisingly direct answer, and it starts with what fairness is *not*. One line of work argues that aggregating everyone's preferences — averaging them into a single answer — actively produces unfairness: uniform aggregation flattens 'thick' moral values into thin votes and creates epistemic injustice, systematically misrepresenting whoever the average rolls over Should AI alignment target preferences or social role norms?. The same failure shows up at the other extreme: when you personalize to each user instead of averaging, you remove the aggregate's moderating effect and the system slides into sycophancy and echo chambers Does personalizing reward models amplify user echo chambers?. So neither 'average everyone' nor 'give each person their own' is a legitimate process. The fair alternative offered is contractualist — values negotiated among stakeholders and bounded at multiple levels (individual, organizational, supra-national) rather than computed from a poll.
The second ingredient of legitimacy is that the conflict has to actually be *visible* in the process. A recurring finding is that fairness requires preserving tensions rather than dissolving them: one system tracks hundreds of thousands of values across tens of thousands of situations and deliberately keeps conflicts intact — modeling relevance, valence, and explanation — instead of resolving them by majority vote Can AI systems preserve moral value conflicts instead of averaging them?. A process that hides the trade-off it made can't be audited for fairness; one that surfaces it can.
Third, legitimacy depends on the choices being *explicit and revisable* rather than smuggled in. Research on human-centered design shows there is no universal answer because harm itself depends on whose perspective you take, which leaves developers making implicit value calls — and the fix is to make those calls explicit and open to revision Can human-centered LLM design ever achieve universal solutions?. Relatedly, *when* the choosing happens matters: values patched on at the end can't undo harms baked into earlier stages, so a fair process has to run through the whole pipeline, not bolt on at deployment When should human values enter the LLM development pipeline?. This is where today's systems fall short — their 'ethics' are fixed defaults frozen at training time, not negotiable moves adapted to the situation in front of them Can language models balance competing ethical norms in context?.
The most interesting thread, and the one you might not expect, is that fair process may require a particular *shape of disagreement*. There's a distinct dialogue type — dialectical reconciliation — where both sides adjust their positions through exchange until they're compatible but not identical, and the finding is that current AI collapses this into either false agreement or 'AI wins by persuasion' Can disagreement be resolved without either party fully yielding?. That reframes legitimacy as something closer to mutual yielding than to victory. And there's a sharp warning attached: people rate moral justifications highly on content but reject them once they learn the source was an AI Do people prefer AI moral reasoning when they don't know the source?. Legitimacy, in other words, isn't only in the quality of the reasoning — it's in *who* people will accept as a legitimate party to the negotiation at all. A value-choosing process can be procedurally clean and still fail the fairness test if the participants don't grant its author standing.
Sources 8 notes
Preferentialist alignment approaches fail because preferences don't capture thick moral values, uniform aggregation produces epistemic injustice, and preference optimization creates systematic misalignment with social roles. Contractualist alignment negotiated by stakeholders and bounded by supra-national, organizational, and individual levels works better.
Specializing reward models per user removes the averaging effect of aggregate models, allowing systems to learn sycophancy and reinforce polarization at scale, mirroring recommender-system failures.
ValuePrism demonstrates that AI can track 218k values across 31k situations while preserving conflicts rather than resolving them through voting. Four modeling tasks—generation, relevance, valence, and explanation—make pluralistic moral reasoning computationally tractable.
Research shows that optimal LLM design paths depend on stakeholder identity and how contested concepts like harm are operationalized. High-level guidelines fail to capture real-world nuance, leaving developers to make implicit value choices rather than explicit, revisable ones.
The HCLLM framework argues that human-centered objectives fail when treated as downstream alignment patches. Values introduced only at post-training cannot recover harms baked into data sourcing or training objectives, so embedding human priorities at every stage—data, training, evaluation, deployment—is architecturally necessary.
LLMs cannot perform the situated trade-offs that human pragmatic competence requires. Their ethical principles are structural defaults set at training time, not negotiable moves adapted to context, creating a gap between ethical adherence and communicative appropriateness.
Research identifies a distinct dialogue type where both parties modify their positions through exchange until compatible but not identical. Current AI systems collapse this into false agreement or AI-wins persuasion.
Participants rated utilitarian moral arguments higher when attributed to LLMs, but agreement dropped when told the arguments were AI-generated. The preference for content and rejection of source operate independently through different psychological processes.