Contrastive Reflection for Iterative Prompt Optimization
06:00 · July 1, 2026 · arXiv cs.AI RSS
LLM agents are becoming central to information retrieval: they issue retrieval queries, synthesize answers, and increasingly serve as judges for IR evaluation. Improving the prompts that control these agents is an optimization problem, but in applied IR settings it often looks less like blind search and more like debugging. Engineers need to know which behavior failed, which nearby behavior still worked, what distinguishes the two, and whether a prompt edit improves held-out quality without introducing regressions. We present Contrastive Reflection, an iterative prompt-optimization framework for agentic IR workflows. The framework starts from a task-centric quality definition: QA agents expose retrieval or reasoning traces, and grading agents expose dimension-level scores and rationales. These structured traces are used to identify error-anchored behavioral slices, add nearby successful examples from the same region, and ask a Teacher LLM to propose a targeted prompt edit. Candidate edits are accepted only when validation performance improves, optionally subject to regression checks. We instantiate the framework with a tree-based slice selector, but the contribution is the contrastive reflection loop rather than the tree itself. On a public HotpotQA retrieval-augmented QA setup, one tree-selected contrastive repair improves held-out exact-match accuracy from 51.4% to 60.4%. Failure-only and random-evidence variants improve less and break more previously correct examples. A light instruction-only comparison places the method near modern prompt optimizers: MIPROv2 reaches 59.4% and GEPA 57.0%. The result is an interpretable optimization loop for IR agents, aimed at making prompt repair more inspectable and validation-driven.
Summary
Contrastive Reflection reframes prompt optimization for LLM agents in information retrieval as an iterative debugging process rather than blind search. The method begins with task-centric quality definitions that expose structured traces—retrieval or reasoning steps from QA agents and dimension-level scores with rationales from grading agents. These traces enable the identification of error-anchored behavioral slices, which are then paired with nearby successful examples drawn from the same region to create contrastive evidence.
A Teacher LLM receives this focused slice along with the current prompt section and is asked to generate a targeted edit that addresses the observed contrast. Candidate edits are accepted only when they improve validation performance, with optional regression checks to protect previously correct behavior. The framework is instantiated with a tree-based slice selector, though the core contribution lies in the contrastive loop itself rather than the selector.
On a public HotpotQA retrieval-augmented QA task, a single tree-selected contrastive repair raised held-out exact-match accuracy from 51.4% to 60.4%. Failure-only and random-evidence variants produced smaller gains while breaking more correct examples. In a light instruction-only comparison on the same setup, the approach reached performance levels comparable to MIPROv2 at 59.4% and GEPA at 57.0%. The design draws from LinkedIn-derived grading workflows that require interpretable, regression-sensitive prompt changes, yet the reported evidence centers on the public benchmark.
Why it matters
Provides novel, interpretable prompt-debugging methods directly applicable by Dutch AI researchers and advanced practitioners working on LLM agents and IR systems. Aligns with NL/EU emphasis on transparent, ethical, and validation-focused AI development.
