If an AI grader has small habits, like favoring familiar phrasing, do they snowball when it judges long, multi-step work?

This explores whether an AI grader's slight tilts get amplified when it has to score long inputs or many-step reasoning. The collection has no note that measures a judge's bias against context length or step count. It does have evidence for each ingredient: small tilts that add up, long inputs that make readers worse, and length itself being a shaped and unreliable signal. Together they point toward yes, but the collection doesn't prove it for graders.

The clearest case of a small preference compounding is not about graders at all. LLMs lean toward frequent words, and general words are more frequent than specific ones. So Does word frequency correlate with semantic abstraction? finds that repeatedly choosing the common paraphrase drifts outputs toward abstraction and erases expert-level detail. One choice looks harmless. A consistent direction repeated over many choices becomes a systematic shift. Noise would cancel out, but a directional tilt accumulates. Do large language models develop coherent value systems? suggests model preferences are directional in this way: independently sampled preferences form unified utility functions that get more coherent as models grow. That is the kind of tilt that compounds.

Long context is the first way this could get worse. Does reasoning ability actually degrade with longer inputs? shows reasoning accuracy falling from 92% to 68% with just 3,000 tokens of padding, even with chain-of-thought prompting. A grader reading a long transcript is a reasoner reading long input. It's plausible that as its careful reasoning weakens, its default leanings fill the gap. That is my inference, since the study tested solvers, not judges. Extra thinking doesn't obviously fix it. Does more thinking time always improve reasoning accuracy? found accuracy dropping from 87.3% to 70.3% as thinking tokens grew from about 1,100 to 16K. Why does chain of thought accuracy eventually decline with length? describes the same inverted-U shape. A judge that deliberates at length before a verdict may be on the wrong side of that curve.

Multi-step reasoning raises a second issue: what the grader uses as a shortcut. Does longer reasoning actually mean harder problems? shows that trace length tracks difficulty only on familiar problems and decouples on unfamiliar ones. A grader that reads a long, effortful-looking chain as a sign of quality is trusting a proxy that breaks off-distribution. Can judges that reason about reasoning outperform classifier rewards? shows one way to grade multi-step reasoning: judges that write their own reasoning about each step beat classifier-style scorers with far less training data. Step-level judging is where a tilt could either accumulate or be caught, but that note measures accuracy, not bias.

The nearest direct evidence of a small reward-side preference compounding is in the optimal-length note above. RL training gravitates toward shorter chains as models improve, and the authors describe simplicity emerging from the reward signal rather than from explicit instruction. A slight preference in the grader, applied over many training steps, reshapes the model's behavior. So the collection supports "compounding" over optimization steps and "degradation" over long inputs. It doesn't yet show that the two combine in a single grader. That would be a good experiment: score the same reasoning chain at increasing lengths and see whether the judge's preferences drift.