Rater State Bias in RLHF Preference Data: An Audit Framework
06:00 · July 21, 2026 · arXiv cs.AI RSS
We identify a structured confound in Reinforcement Learning from Human Feedback (RLHF). Pairwise preference labels are intended to reflect the compared outputs, but they may also reflect the rater's state during annotation. Under sustained stressful or distressing conditions, raters' preferences may shift over time. As a result, preference data can encode rater state alongside judgments about response quality. These shifts differ from ordinary disagreement or random label noise. They are state dependent, can be shared across annotators working under similar conditions, and can propagate through reward modeling and policy optimization. We therefore propose rater state shift as a plausible and testable source of structured bias in RLHF preference data. This paper develops a hypothesis and an audit framework for studying this source of bias. We define rater state shift, rater state confound, and correlated rater state bias. We also define survival level emotional authenticity as a measurable response pattern using lexical, pragmatic, discourse, and safety related features. We analyze how correlated rater state bias can survive aggregation and enter learned reward signals. We derive five falsifiable predictions and effect size thresholds for an initial audit. Finally, we present an audit protocol and pilot study plan that can be applied to publicly available instruction tuned models. We do not infer the training history of any specific deployed model. Our goal is to isolate a plausible and testable source of structured bias in RLHF preference data.
Summary
This arXiv paper examines a structured source of distortion in the preference data used for Reinforcement Learning from Human Feedback. It argues that pairwise labels collected from annotators can encode not only comparative judgments of model outputs but also systematic shifts in the rater’s internal state when annotation occurs under sustained stress or distress. These shifts are distinguished from ordinary label noise or stable annotator disagreement because they are time-dependent, can be shared across multiple raters exposed to similar conditions, and therefore survive aggregation.
The authors introduce three interlocking definitions. A rater state shift refers to a measurable change in affect, attention, or regulation that develops during the annotation session itself. When that change alters the recorded preference, the label becomes a rater state confound. When such confounds are correlated across annotators and retained by the learning process, they produce correlated rater state bias in the resulting reward model. The paper traces how this bias enters the reward function during maximum-likelihood training and is then amplified when the policy is optimized against the contaminated signal under a KL constraint.
To make the hypothesis testable, the work supplies five falsifiable predictions together with effect-size thresholds. It also supplies an audit protocol that operationalizes “survival-level emotional authenticity” through lexical, pragmatic, discourse, and safety-related features of the labeled text. The protocol is accompanied by a pilot-study design intended for publicly available instruction-tuned models; the authors explicitly refrain from claiming to recover the training history of any deployed system.
The central claim is therefore methodological rather than diagnostic: if annotation conditions can induce state-dependent label shifts, then RLHF pipelines contain a previously unexamined pathway for structured bias that warrants direct measurement before the signal is treated as a stable proxy for output quality.
Why it matters
Directly supports ethical and transparent AI priorities central to Dutch/EU AI strategy; offers reproducible audit methods that Dutch research teams and advanced practitioners can apply to alignment pipelines and bias evaluation.
