Could the mere timing of model releases make AI progress look faster or slower than it really is?

This explores whether the timing of model releases can bend the trend lines people draw from Elo leaderboards. The corpus has only one note that measures Elo progress directly, so this is a thin area. That note is still a good place to see why the question matters. Epoch AI's analysis finds that open models small enough to run on a consumer GPU gain about +125 Elo per year, against about +80 per year for frontier models. It reads the steeper slope as a sign that the gap is closing Are open models closing on frontier AI faster than expected?. A slope like that depends on which models sit at which dates. If open releases cluster near the end of the window, or a frontier lab ships late in a cycle, the fitted line can tilt without any change in the underlying pace of progress. The corpus doesn't test this sensitivity directly. Treat it as an open question to ask of any slope comparison, not a settled answer.

The less obvious point is that Elo is relative. A model's rating comes from head-to-head votes against whatever else is on the leaderboard at the time. So a release changes more than its own data point: it also changes the pool every other model is measured against. Research on ranking systems shows how this kind of self-referencing data goes wrong. When a system learns from outcomes its own earlier choices shaped, it can lock into patterns that amplify those choices, unless selection bias is modeled explicitly Why do ranking systems need to model selection bias explicitly?. That note is about YouTube recommendations, not Arena voting. The parallel still applies: which models get shown, voted on, and compared is not random, and it shifts with release timing.

A second issue is what a 'model' even means at the moment it's scored. Work on execution harnesses (the scaffolding that runs a model on a task) shows that the same frozen weights can score very differently depending on the system around them. A single reusable runbook lifted DeepSeek-V4 Flash by 5.4 points on Terminal-Bench without any retraining Can execution harnesses lift model performance without retuning weights?. If a model is benchmarked at launch with immature tooling and its successor arrives once tooling has matured, part of the measured jump comes from the ecosystem, not the model. Release timing then decides how much of that tooling credit lands in each data point.

Finally, any single leaderboard snapshot is one draw from a noisy process. Work on deterministic settings makes the general point that getting the same output every time is not the same as reliability Does setting temperature to zero actually make LLM outputs reliable?. Take a slope fitted through a few dozen ratings, each recorded once, at dates set by company launch calendars: it shows a trend, but it shouldn't be read as a precise rate. To probe this properly, the collection would need work on ratings recomputed over time, on tests of how a slope changes when release dates shift, or on Arena's own methods. None of that is in the corpus yet.