Do open-weight AI models like DeepSeek actually beat closed ones like GPT or Claude at any specific kind of task?
Do open-weight models dominate particular task categories over proprietary alternatives?
This explores whether open-weight models (ones whose weights anyone can download, like DeepSeek, Qwen or IBM's Granite) clearly win certain kinds of tasks over closed models like GPT, Claude and Gemini, or whether the picture is more mixed.
This explores whether open-weight models own particular task categories outright, or whether something messier is going on. The short answer from this collection: it has no clean scoreboard of categories where open models win. What it does show is two patterns that may matter more. First, open models take a large share of real-world use even where they aren't the most capable. Second, in narrow, well-defined tasks, targeted training can close the gap with frontier closed models.
The clearest evidence on usage comes from an analysis of more than 100 trillion tokens routed through OpenRouter How much do open-weight models actually get used in practice?. Open models such as DeepSeek and Qwen handled over 30% of that traffic, including in areas where closed models lead on capability. The reason is that users aren't only optimizing for the smartest answer. They weigh cost, speed and control against quality. So 'dominance' may be the wrong frame. The market stays mixed because different jobs value different trade-offs, and the 'best' model on a leaderboard Can crowdsourced votes reliably rank language models? isn't always the one people actually pick.
Where open models do catch up on capability, it tends to happen in tasks with a rigid structure. Function calling, where a model turns a request into a correctly formatted tool or API call, is the standout example. IBM's open Granite-20B model closed much of its gap with GPT, Claude and Gemini by splitting function calling into seven specific skills (spotting which function to use, filling in parameters, chaining calls, and so on) and training on each one explicitly Can breaking function calling into subtasks improve model generalization?. Separately, small models trained with DPO on correct and incorrect examples produced by a large teacher model got very good at function calling. Seeing the wrong answers helps them avoid the formatting mistakes that ordinary fine-tuning leaves behind Can small models match large models on function calling?. The same thing appears outside open-vs-closed comparisons too: Walmart's small in-house ranking models beat the LLM that taught them once they were trained on enough labeled search queries Can smaller models outperform their LLM teachers with enough data?. The pattern seems to be that the narrower and more checkable the task, the less raw frontier scale matters.
One point you might not have expected: open weights matter for understanding models, not just for using them. A study of RL post-training found it quietly reinforces one output style from pretraining and suppresses the others. That effect is largely hidden when researchers start from proprietary models, because they can't see what went into those models Does RL training collapse format diversity in pretrained models?. So one area where open models clearly lead is research transparency.
If you want a firm ranking of open-weight wins by task type (coding vs. creative writing vs. chat, for example), this collection doesn't have it yet. The honest summary: open models are widely used because they're cheaper and more flexible, they can match closed models on narrow structured tasks with the right training, and they're the only option when you need to see inside the model.
Sources 6 notes
Analysis of 100+ trillion tokens on OpenRouter found open models like DeepSeek and Qwen captured large usage shares even where closed models lead on capability. Users trade cost, latency, and other factors against capability, creating persistent market heterogeneity.
Chatbot Arena's 240K+ crowdsourced preference votes produce credible model rankings because the underlying questions are diverse and discriminating, and crowd judgments correlate with expert raters—validating human preference as a scalable evaluation signal.
Granite-20B-FunctionCalling shows that explicit training across seven granular subtasks—nested calls, chaining, parallel functions, name detection, parameter detection, next-best function, and response generation—generalizes better than umbrella datasets like ToolLLM. This multi-task approach closes the performance gap with GPT, Claude, and Gemini.
Small models fine-tuned via DPO on correct and incorrect function-calling examples from a large teacher model achieve high accuracy on logical and mathematical tasks. DPO's explicit negative examples directly target the rigid output format failures where SFT alone underperforms.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
Show all 6 sources
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Improving Small-Scale Large Language Models Function Calling for Reasoning Tasks
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- The Latent Role of Open Models in the AI Economy
- The Art of Scaling Reinforcement Learning Compute for LLMs