Can reference examples make LLM judges reliable enough for self-improvement?

The paper asks whether "reference-guided LLM-evaluators can bridge" the gap between RLVR, which needs a ground-truth verifier, and alignment tuning, where none exists — by serving as "soft verifiers." Measured across 11 LLM-judges in pairwise comparison, prompting with a reference answer generated by a stronger model (GPT-4o) and explicit instructions on how to use it produced "a 6.8% absolute improvement over the reference-free baseline"; the paper notes that "naive incorporation of references without explicit guidance... yields only modest improvements." Built on these improved judges, a two-stage training run — SFT distillation on high-quality references, then DPO using the model itself as judge — beat SFT distillation alone by "+20.2 / +17.1 points" on AlpacaEval/Arena-Hard for Llama-3-8B-Instruct, beat reference-free self-improvement by "+5.3 / +3.6 points," and reached "performance comparable to training with ArmoRM, a strong finetuned reward model."

The mechanism is three pairwise prompting strategies: Ref-Free, a strong baseline judging instruction-following, factuality, and verbosity without any reference; RefEval, which instructs the judge to assess "which candidate output more closely aligns with the quality and content exemplified by the reference" while still addressing the instruction; and RefMatch, which pushes further, telling the judge its "goal is to determine which output demonstrates closer similarity to the reference." The judges are never retrained — only prompted differently — and then plugged into DPO as the sole supervision signal, which the paper calls self-improvement because "no external human or AI feedback is required."

This is a different route to the same bridge that Can reasoning during evaluation reduce judgment bias in LLM judges? builds: J1 trains the judge itself with RL on synthetic high/low-quality pairs so it reasons before scoring, manufacturing verifiability inside the judge. This paper never touches the judge's weights — it anchors an unchanged LLM-as-Judge to an external reference answer and lets the anchor do the work RL does elsewhere. It also bears directly on Where should an LLM judge sit in an optimization loop?: the reference-guided judge here holds exactly that authority, supervising a full DPO run end to end, and the paper reports only accuracy gains on held-out benchmarks, not whether reference-anchoring shrinks the error set an optimizer could exploit or simply relocates it. It also sits beside Can LLM judges be fooled by fake credentials and formatting?: both treat judge manipulability as the thing to fix, but this paper's fix is an external reference anchor rather than bias-resistant training of the judge.

The excerpt does not report the judges' accuracy in absolute terms (only the 6.8-point delta), so it is unclear how exploitable the reference-guided judge remains in isolation, and it does not test whether reference-guided self-improvement is more robust to judge gaming than reference-free self-improvement — only that it scores higher on AlpacaEval and Arena-Hard, which are themselves LLM-judged. The paper also flags its own limit: reference-guided gains were weaker on Creative Tasks for Qwen2.5-7B-SFT than for Llama-3-8B-Instruct, which it attributes to open-ended tasks needing "more extensive post-training" to use references well. The implication is that a reference narrows what "better" means during self-improvement and that narrowing measurably helps, but whether it closes exploitable gaps in the judge or just moves them remains untested here.