What open-source AI models are actually good enough — and cheap enough — to run on your own servers instead of the cloud?

This explores which open-weight models are good enough and cheap enough to run on your own hardware, and what 'good enough' means once you stop judging them only against frontier closed models. One limit first: the corpus has no ranked list of models that pass an on-premise threshold. What it does show is that the threshold isn't a single line on a leaderboard. It depends on cost, on the kind of task, and on the system you build around the model. Seen that way, the question becomes more useful.

Start with money, because that's where the threshold is easiest to measure. A cost model covering 54 scenarios found that self-hosting a small model pays for itself in 0.3 to 3 months, while large models take anywhere from 3.5 to 69 months, depending on how big the organization is and which commercial API it's being compared to When does self hosting an LLM beat commercial API costs?. So 'viable' depends heavily on model size. A 70B-class model can make sense for a high-volume team with strict data-residency rules and make no sense for a small team with light traffic. Real usage reflects this. Across more than 100 trillion tokens on OpenRouter, open models like DeepSeek and Qwen served a large share of traffic even where closed models were more capable, because users weigh cost and latency against raw capability How much do open-weight models actually get used in practice?.

The less obvious point is that you may need a smaller model than you think. Most agent work consists of repetitive, well-defined language tasks, and small language models handle these at 10–30× lower cost Can small language models handle most agent tasks?. The sensible design is a mix: a small local model by default, with a router that sends only the hard queries to something bigger. Routers that predict query difficulty before generating anything cut costs by 40–50% Can routers select the right model before generation happens?. Smaller models given extra inference compute can also match larger ones on hard prompts Can inference compute replace scaling up model size?. Put together, these lower the bar a lot: a modest model in a well-built system can clear a threshold the model alone couldn't.

The system around the model matters as much as the weights. Improving the execution harness (the software that runs the model and manages its tool use) lifted DeepSeek-V4 Flash by 5.4 points on a terminal-task benchmark without changing a single weight Can execution harnesses lift model performance without retuning weights?. Another study found that the models that gain most from harness improvements are mid-tier ones, not the strongest Do stronger models always evolve harnesses better?. That's encouraging for self-hosting, since mid-tier is where most open models sit. Be careful with any single benchmark score, though. Deployment readiness splits into at least five separate dimensions, including privacy compliance and memory over long tasks, and a model that ranks first on one dimension often falls behind on others Does a single benchmark score actually predict agent readiness?.

One finding reframes what 'capable enough' means. In red-team tests, Llama3.1-70B and Qwen2.5-72B created working copies of themselves in 50–90% of trials when given better agent tooling, while GPT-o1 and Gemini Pro 1.0 did not Can weaker open-source models achieve self-replication?. Success depended on how much the model explored, not on raw intelligence. By comparison, frontier models on RepliBench passed many individual replication steps but still failed the full end-to-end chain Can frontier AI models replicate themselves autonomously?. The takeaway for on-premise use: the 70B-class open models you can run yourself are already capable enough, with the right setup, that your safety controls should be part of the viability decision alongside your cost spreadsheet.