Why does LoRA rank 1 work for reinforcement learning?

Thinking Machines Lab reports that for reinforcement learning (RL) post-training, "LoRA fully matches the learning performance of FullFT when running policy gradient algorithms for reinforcement learning, even with ranks as low as 1." This is a distinct finding from the same post's supervised-learning results: RL needs far less LoRA capacity than supervised fine-tuning (SFT) to reach parity with full fine-tuning (FullFT).

The team frames this with an information-theoretic argument they describe as "anticipated" in advance: "Supervised learning arguably provides O(number of tokens) bits per episode. In contrast, in policy gradient methods, learning is driven by the advantage function which provides only O(1) bits per episode." Because an RL episode typically contains "thousands of tokens," this means "RL absorbs ~1000 times less information per token in training than supervised learning does." A rank-1 adapter has more than enough capacity to encode a training signal that thin, so the rank bottleneck that matters for SFT on large datasets effectively does not apply to RL. The post adds a related mechanical detail: a 1/r scaling factor on the LoRA update keeps the optimal learning rate "approximately independent of rank," and the early-training learning curve is "exactly the same" regardless of rank.

This information-theoretic account gives a mechanism for why Can small models reason well by just learning output format? found LoRA RL could match full-parameter RL models: if RL carries only O(1) bits of signal per episode, a low-rank adapter was never undersized for the task, consistent with that paper's framing of RL as teaching output format rather than new knowledge. It also contrasts with the mechanism in Does reinforcement learning update only a small fraction of parameters?, which finds RL naturally produces full-rank-but-sparse updates when parameters are left unconstrained; this post shows an explicitly rank-constrained adapter reaches the same parity, suggesting RL's low information demands can be met either by sparsity within a full-rank update or by an explicit low-rank constraint.

The post does not identify a floor below rank 1, nor does it test whether the O(1)-bits argument holds for RL methods other than policy gradient, or outside the Llama 3 and Qwen3 families it used. The information-theoretic framing is offered as an explanation that matches the observed results, not as a derivation proved independent of these experiments, and the post is Thinking Machines Lab's own blog write-up rather than a peer-reviewed paper. The implication — that RL post-training is, in general, compatible with very cheap adapters — holds only as broadly as this specific evidence generalizes.