Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation
06:00 · August 17, 2026 · arXiv cs.AI RSS
Evaluating language-model agents at scale increasingly relies on a second language model as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time. Such a judge is a reward-free proxy whose value depends on whether it can be trusted, yet existing judges either hand-write the scoring rubric, as in G-Eval, or fine-tune the judge's weights, and both tend to credit fluent but unsuccessful trajectories as successes. We instead induce the text of an agent-judging rubric from a small set of ground-truth-labeled trajectories, grounding it in true outcomes. We present RubricForge, which evolves a judge rubric by reflective evolution against labeled trajectories to maximize agreement with the environment reward, freezes it, and applies it to held-out trajectories in one model call with no environment access. The optimized artifact is human-readable text, so every verdict is attributable to named criteria. Using one frozen 7B model as both agent and judge, on tau-bench (173 labeled trajectories drawn from 220 rollouts) and WebShop (160), the principal gain is faithfulness rather than raw agreement. The edge over a generic G-Eval judge is not statistically significant (McNemar p = 0.248), and absolute-score calibration marginally favors the generic judge (|err| difference -0.048, p = 2x10^-4). Yet RubricForge over-credits failed trajectories roughly half as often (0.115 vs. 0.173 false-pass rate on tau-bench, with three over-credit catches and zero reversals) and ranks graded WebShop outcomes more faithfully (Spearman 0.410 vs. 0.370). For a reward-free evaluator the false-pass rate, not aggregate agreement, is the deployment-relevant quantity, since a false pass ships a broken agent whereas a false fail merely costs a retry.
Summary
Evaluating language-model agents at scale often requires an automatic judge because executable environment rewards are costly or unavailable once an agent moves into deployment. Existing LLM-as-a-judge methods either rely on hand-written scoring rubrics, as in G-Eval, or fine-tune the judge model itself; both approaches tend to accept fluent but unsuccessful trajectories as successes. RubricForge addresses this by inducing the rubric text itself from a modest set of ground-truth-labeled trajectories, using reflective evolution to align the rubric with actual environment outcomes rather than surface features.
The process keeps the underlying judge model frozen. A small collection of labeled rollouts supplies the supervision signal; the evolving rubric is scored against the known environment reward, refined, and then frozen. At evaluation time the rubric is applied to held-out trajectories in a single model call with no further environment access. Because the artifact remains human-readable text, each verdict can be traced to explicit criteria that practitioners can inspect or edit.
On tau-bench and WebShop the principal improvement appears in the false-pass rate rather than raw agreement. With a single 7B model serving as both agent and judge, RubricForge reduced over-crediting of failed trajectories from 0.173 to 0.115 on tau-bench while producing three additional catches and no reversals relative to a generic G-Eval judge. Graded WebShop outcomes were ranked more faithfully by Spearman correlation, although absolute score calibration remained marginally better for the hand-written baseline. The authors note that, for a reward-free evaluator, the false-pass rate is the deployment-critical quantity: a false pass ships a broken agent, whereas a false fail only triggers a retry.
Why it matters
This research is highly relevant for Dutch AI researchers and enterprises focused on developing trustworthy and transparent AI systems. By reducing false-pass rates in automated agent evaluation, it provides a robust methodology that aligns with the EU's stringent requirements for AI reliability and safety.
