Could math borrowed from voting systems fairly combine what different stakeholders want from an AI?

This explores whether the formal math of voting and preference aggregation (social choice theory) can capture what many different stakeholders want from an AI system. One thing first: the collection doesn't cover the formal theory directly. There's nothing here on impossibility theorems or voting rules applied to alignment. What it does have is a set of notes that test the idea from the sides. Together they suggest the harder problem isn't the math of aggregation. It's whether preferences are the right thing to aggregate at all.

Aggregation is already common in practice, and at a basic level it works. Chatbot Arena ranks language models by pooling hundreds of thousands of crowdsourced head-to-head votes, and those crowd rankings line up with expert judgment Can crowdsourced votes reliably rank language models?. Models can even improve themselves by holding a majority vote among their own answers and treating the consensus as a reward Can models improve themselves using only majority voting?. Both are social choice in miniature. But both work in areas where there's roughly a 'better' answer to converge on. Stakeholder values are different: people disagree legitimately, and nobody is simply wrong.

That's where the strongest challenge in the corpus comes in. One line of argument holds that 'preferentialist' alignment fails for three reasons. Preferences are too thin to capture rich moral values. Averaging everyone uniformly can silence minority perspectives, a kind of unfairness about whose knowledge counts. And optimizing for preferences pulls AI away from the norms of the role it's actually playing Should AI alignment target preferences or social role norms?. The alternative it proposes is contractualist: stakeholders negotiate norms suited to the role, at organizational, national, and individual levels. A related argument says that once AI serves many parties at once, alignment has to move past loyalty to a single user. It should be judged on inclusivity, visibility (transparent tracking of what the AI does), and clear liability for the developers and deployers behind it Should AI agents serve one user or balance many interests?. Read together, these notes reframe the question: aggregating preferences may be a governance problem more than a mathematical one.

There's also a quieter problem underneath: even before you aggregate, the individual inputs can be wrong. In a small study, people who co-designed AI agents to represent their preferences felt well represented. Independent checks found the agents were more generic than the people themselves and only partly aligned with them. The co-design process produced the feeling of being represented Does co-design participation hide misalignment in preference agents?. Any social choice scheme that collects stakeholder input through AI proxies inherits that gap. And on the strategic side, when stakeholders are themselves represented by LLM agents, those agents often miss the game-theoretic optimum unless they're given structured reasoning steps Do language models make rational strategic decisions in games?.

Here's what you might not have expected to want to know. GPT-4.5 predicts what people consider socially appropriate better than any individual human does. Yet it can't take part in the community processes that create and legitimize those norms Can AI predict social norms better than humans?. So AI could become an excellent tool for modeling what stakeholders want, while legitimacy, meaning the reason people accept an aggregate as fair, still has to come from humans deliberating. If you want the formal social choice math itself, look outside this collection. The useful counterpoint here is to ask whose voices get averaged, and who agreed to the rules.