Can aggregated LLM rankings match human expert judgments?

Doshi, Bell, Mirzayev, and Vanneste (Strategic Management Journal, 2025) tested whether generative AI can evaluate strategic decisions — specifically, ranking business models by likelihood of success — as well as human experts do. Across two studies of 60 business models each (one AI-generated, one submitted to a competition), they had multiple LLMs, assumed LLM roles, and prompts perform pairwise comparisons, then compared the rankings to those of human experts. Their finding: "generative AI often produces evaluations that are inconsistent and biased. However, when aggregating evaluations, AI rankings tend to resemble those of human experts." The managerial summary is blunt about the implication: "Generative AI's single evaluations are often inconsistent or biased," but aggregating "many evaluations across LLMs, prompts, or roles" produces results that "resemble those of human experts."

The paper explains the gain through two mechanisms borrowed from wisdom-of-crowds research: a diversity effect (aggregating across different LLMs, roles, and prompts while holding the number of predictions constant) and a scaling effect (aggregating all predictions regardless of source). Using seven LLMs — Claude2, PaLM2, Gemini Pro, Llama2, GPT-3.5, GPT-4, and GPT-4 Turbo — across 10 roles and 2 prompts, they found "both studies show modest increases from diversity and substantial increases from scaling," concluding that "in these two contexts, gains from aggregation are achieved primarily through scaling." The authors frame this as extending research on aggregating human predictions (Csaszar & Eggers; Knudsen & Levinthal) to a new population of evaluators: artificial ones, aggregated "by LLM, role, and prompt" rather than by human characteristics like skill or cognitive style.

This complicates the pairing of Can AI forecasters beat expert humans at venture evaluation? with Do LLMs consistently favor the same strategic choices regardless of context?. The venture tournament found pooling AI with humans adds nothing over the single best model — but that study pools AI and human predictions together, whereas Doshi et al. pool AI evaluators among themselves and find real gains from doing so, mostly via scale. Trendslop is the harder complication: Romasanta et al. found six different LLMs converge on the identical recommendation for a given tension, a shared, correlated bias that reordering options moved more than business context did. Aggregation cancels inconsistency when errors are independent across evaluators; it is unclear it would cancel a bias that most evaluators share for the same reason (shared training-data skew toward certain vocabulary), which Doshi et al.'s pairwise-ranking task may simply not have triggered.

The excerpt's ground truth is agreement with human experts, not realized business outcomes — unlike the venture tournament's use of actual fundraising results, nothing here confirms that AI-aggregate rankings predict which business models actually succeed, only that they converge toward what human experts would rank highest. The LLMs tested (GPT-4, Claude2, PaLM2, Gemini Pro, Llama2) are from 2023, well before current frontier models, and the task — ranking short, AI-generated or competition-submitted business model descriptions — is narrower than high-stakes, irreversible strategic decisions like M&A or market entry that the paper's introduction frames as the ultimate target. The authors themselves speculate, rather than establish, that the diversity/scaling split "depends on the nature of the evaluation task," naming information completeness and uncertainty-resolution timing as candidate moderators still untested.