Research note: Evidence on AI R&D Progress from NanoGPT

Source: Manish Shetty, METR · 2026-04-21

I.

We want to measure and understand how much AI agents can accelerate AI R&D and how this is changing over time. There are various sources of evidence we can look to here, including anecdotes about autonomous contributions (AlphaEvolve and TTT-Discover speeding up a GPU kernels, autoresearch yielding speedups in nanochat), progress on benchmarks, and uplift measurement (see our recent post for a longer discussion).

Let’s look at one such leaderboard: the nanogpt speedrun. The goal is to train a language model to a target validation loss on FineWeb using 8×H100 GPUs as fast as possible. It’s a small-scale version of LLM pretraining with a public history of contributions, with four recent ones credited to AI agents as of April 2026. The optimization activities map to pretraining research such as architecture changes, writing kernels, and improving optimizers. Contributions, such as the Muon optimizer, have made it to frontier-scale models like GLM-4.5 and Kimi K2.

However, there are some challenges to interpreting this evidence:

Contamination.

Survivorship bias. We only (directly) see ideas that worked. Humans and agents may have tried ideas that failed, and those don’t show up in the record.

Scale-dependence. Training GPT-2 on 8×H100s is not frontier pretraining.

Composability. Pretraining is one piece of the AI R&D stack. How acceleration on it composes with other parts is an open question, though tasks like posttrainbench could help cover other pieces.

The speedrun has two tracks: a small track (target loss 3.28, starting from GPT-2-small 124M params) and a medium track (target loss 2.92, starting from GPT-2-med 350M params). This post focuses on the small track.

From May 2024 to March 2026, 36 contributors have submitted 77 records — each one a new version of the training code that beats the previous best time — cutting the training time from 45 minutes to 1.43 minutes, a 31x speedup.3 Every record has a PR/commit with diffs, descriptions, and often cited papers or tweets describing the idea.

Humans have made lots of progress, including deep ideas and breakthroughs.

Relatively shallow contributions have had a big impact. Shallow and moderate contributions together drove roughly 21x of the 31x. For example, record 12 adopted the newly released FlexAttention, a PyTorch API for efficiently implementing attention patterns, and got a 30% speedup. The library itself is sophisticated, but the contribution was essentially migrating to it.

Deeper ideas appeared throughout, not just early on. Record 3 introduced the Muon optimizer, original research on Newton-Schulz orthogonalization, later adopted widely including by Kimi K2 and GLM-4.5. Late in the speedrun, record 58 invented Paired Head Attention, a novel attention mechanism with no clear precedent, and record 62 invented Bigram Hash Embedding, uniquely combining a 2017 hash embeddings idea with ideas from DeepSeek’s Engram.

Early progress was largely about catching up to the frontier research; later records increasingly invented new ideas.

Early records (first 20 records, May 2024 – Jan 2025) were mostly imported or adapted to catch up to frontier research, applying techniques like RoPE, ReLU2, QK-norm, and FlexAttention.

Later records (Jan 2025 – Mar 2026) increasingly invented new ideas: 33% were invented, compared to just one (Muon) in the first 20.

Also, in terms of cumulative speedup: imported ideas drove 6.7x of the 31x total, adapted ideas 3.0x, and invented ideas 1.6x.

Records targeted many layers of the AI stack, and focus has shifted over time.

Above, each row shows a different layer of the model stack, with bar height indicating the relative speedup of each record. Despite NanoGPT being a single chunk of AI R&D (pretraining), records span many layers.

Early records were dominated by model architecture changes. The middle phases shifted toward attention mechanisms and parallelization. In later records, optimizer and kernel-level improvements have picked up.

AI agents have recently contributed, but their current contributions appear relatively shallow.

Between late 2025 and early 2026, four records in the official record history credit an AI agent alongside a human co-contributor. All four are bespoke AI agent systems built by specific teams for ML research and optimization, not general-purpose coding assistants like Claude Code:

Hiverge from Hiverge AI optimized distributed training and skip connection gating.

Locus from Intology fused a softcapped cross-entropy kernel.

Aster from Aster Lab improved kernel memory access patterns.

Station from Dualverse AI increased the LR floor and added a short-sequence curriculum.

All four are real improvements, but based on my analysis none reached the deep or breakthrough end of the scale. However, this isn’t strong evidence that agents produce fewer deep ideas than humans. With four records with no disclosure of inference compute spends, these appear similar to more recent human contributions.

A notable contribution was by Station, which discovered that a config parameter (window_size) had been silently ignored by compiled code, a bug humans had missed for months.

Because each record is co-credited to a human, we also don’t know how human-directed these agent runs were or how many failed attempts preceded the successfully recorded ones. Nor do we know how much compute these agent runs consumed. So these contributions are hard to interpret on their own.