Can online LLM feedback improve direct preference optimization during training?

Direct alignment from preferences (DPO, IPO, SLiC) is attractive because it skips the separate reward model and updates the policy directly from pairwise preferences. But its preference datasets are collected ahead of training and never updated, and the responses usually come from a different model — so as the policy evolves, alignment becomes inevitably off-policy and prone to overfitting. OAIF's fix is simple: on each training iteration, sample two responses from the current model and prompt an LLM annotator to pick the preferred one, supplying online feedback. Despite its simplicity, human evaluation shows OAIF beats both offline DAP and RLHF, and it mitigates reward over-optimization — the overfitting that plagues offline DAP.

Two keepers. First, the online vs offline distinction matters more than the choice among DAP variants: OAIF improves DPO, IPO, and SLiC alike, isolating on-policy feedback as the lever. Second, the AI annotator's feedback is controllable via instruction prompts — you can steer the alignment target by changing how you ask the judge to choose.

This connects the vault's alignment-method thread to the LLM-as-judge thread. The controllable AI annotator inherits the risks documented in Can LLM judges be fooled by fake credentials and formatting? — an online judge that is biased steers the policy toward those biases — and the on-policy framing rhymes with Can agents learn from failure without updating their weights? in treating fresh, current-model feedback as the signal that matters.