AI rankings can match expert rankings of business ideas — but does matching experts mean matching reality?

This explores whether agreement between AI and expert rankings of business models tells us anything about which businesses actually succeed. The short answer from the corpus: agreement with experts has been shown, but agreement with real-world outcomes has not. These are two separate tests, and the collection only really runs the first one for business strategy.

The agreement finding is solid. When many LLM evaluations of strategic decisions are pooled together, the combined ranking looks a lot like the ranking human experts produce, even though any single LLM run is inconsistent and biased Can aggregated LLM rankings match human expert judgments?. Most of the gain comes from running more evaluations, not from mixing different models or prompts. Averaging reduces the noise around a shared signal. What the study doesn't do is check that signal against what happened to the businesses afterward. If experts are systematically wrong about a type of venture, an LLM ensemble that converges on expert opinion will be confidently wrong in the same way. Agreement shows the AI has learned expert taste. It doesn't show the AI can predict outcomes.

The best evidence on actual outcomes comes from a different setting: venture capital. On VCBench, several LLMs predicted which founders would succeed better than human investors did, with one model reaching six times the precision of a market index Can language models beat human venture capital experts?. The useful lesson is the condition behind that result. LLMs cleared the bar because expert investors only modestly beat chance in the first place. This flips the question. In domains like early-stage business prediction, where expert judgment is weak, matching the experts may be a low ceiling rather than a gold standard. An AI that simply copies expert rankings would inherit their mediocrity.

The corpus also suggests why success is hard to predict from any ranking of a business model on paper. Work on AI agents shows a recurring gap between doing well on evaluations and creating real economic value: systems win contests but stumble on long, messy professional work, because the field optimizes what it measures Why do agent benchmarks not predict real economic value?. A separate historical analysis finds that capable products stall when surrounding conditions are missing, such as social acceptance, trust, and standards Why do capable AI agents still fail in real deployments?. Business models fail for the same reasons, and none of those factors show up in a static ranking. There is also a feedback problem. Ranking systems that learn from their own past choices drift toward reinforcing what they already favored Why do ranking systems need to model selection bias explicitly?. If investors start funding whatever AI-plus-expert rankings prefer, those businesses get capital and attention, and their success could make the rankings look predictive when the rankings partly caused it.

One more caution about trusting AI-expert agreement. Evidence from AI replications of marketing experiments shows they reproduce strong findings well but are unreliable on marginal ones Can AI personas reliably replicate human experiment results?. Rankings are likely sturdiest where the call was already obvious and weakest in the close cases where a forecast would actually help. Expert standing also comes from a track record that a community has checked over time, which AI rankings haven't built yet Can AI ever gain expert community trust through participation?. The open question the corpus leaves you with: in fields where experts are barely better than chance, should AI be trained to agree with experts at all, or should it be scored against outcomes instead?