Distributionally Robust Listwise Preference Optimization
06:00 · July 3, 2026 · arXiv cs.AI RSS
Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level. We instead study listwise preference optimization under ranking-label uncertainty: given a prompt and a candidate list, the observed ranking over that list may be ambiguous due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model noise. We propose a pointwise total-variation robust Plackett--Luce objective that directly robustifies the ranking label conditional on the candidate list. The robust loss admits an exact decomposition into the nominal PL loss plus a worst-case PL correction, and the worst-case ranking is obtained by sorting current implicit scores in ascending order, reducing the inner maximization from $K!$ enumeration to $O(K\log K)$. This tractable structure yields strong offline and online optimization guarantees. In the offline fixed-list setting, the robust objective is convex and projected stochastic subgradient reaches global $\epsilon$-suboptimality with $O(\epsilon^{-2})$ sample complexity. In the online policy-induced setting, where candidate lists are generated by the current policy, we establish weak convexity and $\widetilde O(\epsilon^{-2})$ Moreau-envelope stationarity. Experiments in offline LLM alignment show that the proposed robust correction largely preserves performance under clean labels and improves robustness under noise. In online alignment, it makes reward-model-ranked candidate expansion more reliable and improves both reward-model and external GPT-4 judge metrics.
Summary
Existing robust preference optimization methods for language-model alignment have focused primarily on pairwise supervision, applying robustness at the level of datasets, prompts, or individual preference pairs. This work shifts attention to listwise settings, where a prompt is paired with an ordered list of candidate responses and the observed ranking may itself be uncertain because of annotator inconsistency, near-ties, lossy feedback, or noise from reward models. The authors introduce a pointwise total-variation robust Plackett–Luce objective that treats the ranking label as the object of distributional uncertainty while conditioning on a fixed candidate list.
The resulting robust loss admits an exact decomposition into the nominal Plackett–Luce term plus a worst-case correction. Because the inner maximization over permutations is solved simply by sorting the model’s current implicit scores in ascending order, the computational cost drops from factorial to O(K log K) per list. When the list length is two the formulation recovers standard Bradley–Terry or DPO objectives; for longer lists it preserves genuinely listwise structure rather than reducing to independent pairwise corrections.
Theoretical analysis covers both the offline fixed-list regime and the online policy-induced regime. In the offline case the objective is convex, and projected stochastic subgradient descent reaches global ε-suboptimality with O(ε⁻²) sample complexity. In the online setting the objective is weakly convex; an explicit ascending-sort Clarke-subgradient oracle yields Õ(ε⁻²) stationarity measured on the Moreau envelope.
Empirical results indicate that the correction functions as a conservative regularizer on ranking labels. Under clean supervision it maintains performance comparable to the nominal objective, while under structured label noise it improves stability. In online alignment loops, where candidate lists are ranked by a reward model, the robust formulation makes expansion to larger lists more reliable and raises both reward-model scores and external GPT-4 judgments.
Why it matters
This research is highly relevant for Dutch AI researchers and NLP practitioners focusing on LLM alignment and robust AI systems. Improving the reliability of preference optimization aligns well with the EU's emphasis on trustworthy and transparent AI, making it actionable for local enterprises developing compliant language models.
