Top AI leaderboards favor closed models, so why do open, downloadable ones still win so much real-world usage?

This explores why open-weight models win real usage even though closed models lead the leaderboards. The short answer from the corpus is that a benchmark score answers one narrow question, and people choosing a model are asking several others. The clearest evidence comes from OpenRouter's study of more than 100 trillion tokens of traffic. Open models like DeepSeek and Qwen served over 30% of tokens, including in areas where closed models were more capable. Users were weighing cost, speed, and other practical factors against raw capability, and the market didn't settle on a single winner. Instead it stayed split, with different models winning different jobs How much do open-weight models actually get used in practice?.

Part of the explanation is what a benchmark number actually measures: how one model performed under one fixed set of test conditions. The number leaves out much of what happens around the model What do benchmark scores actually reveal about model containment?. That matters because the surrounding system can change results a lot. Better execution setups ('harnesses') raised the scores of frozen models on Terminal-Bench without changing a single weight, and lifted a budget open model, DeepSeek-V4 Flash, by 5.4 points Can execution harnesses lift model performance without retuning weights?. Another design gives a model layered external memory, such as a persistent workspace, helper agents, and saved history on disk, so it can work on longer problems than its context alone allows Can external state caches let models solve harder problems?. In one striking case, a strong model wrote the harness for a weaker one, moving its shaky reasoning into deterministic code. That nearly doubled the weaker model's scores on Theory-of-Mind tests, with no retraining Can a stronger model lift a weaker one at test time without retraining?. So a team can use a frontier model once to design the workflow, then run a cheap open model inside it at scale.

Cheap models also have a way to match expensive ones without any outside help, but only under a specific condition. Running many attempts from a weaker model raises the odds that one of them is right. Matching a strong model reliably, though, requires a way to check answers, such as unit tests, proofs, or type checks, to pick the correct attempt out of the pile When can weak models match strong model performance?. That helps explain why open models do well in tasks like coding, where answers can be checked automatically. Specialization pushes the same way. Walmart distilled an LLM's search-ranking judgments into small BERT models, and with enough augmented training data the small models beat the LLM that taught them Can smaller models outperform their LLM teachers with enough data?. For a narrow, high-volume job, being a generalist on the leaderboard is often less useful than being small, fast, and tuned to the task.

The gap is also shrinking. Epoch AI found that open models small enough for a single consumer GPU are gaining LM Arena Elo about 56% faster than frontier models, roughly +125 versus +80 points per year Are open models closing on frontier AI faster than expected?. The insight you might not expect is that capability is not a fixed property of the weights. It comes from the weights plus the harness, the checkers, and how well the model fits the task. Open models win when the surrounding system can make up the difference, and the corpus suggests that system is improving faster than the gap between models is closing.