CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models
06:00 · August 6, 2026 · arXiv cs.AI RSS
Vision-language systems combine images with retrieved text, but these sources can disagree or jointly fail to support an answer. Reliable models must identify the trustworthy source and abstain when neither is adequate. Existing post-training objectives score instances independently and therefore do not enforce coherent behavior under counterfactual evidence changes. We introduce CARGO-VL, a group-relative framework that optimizes matched variants covering aligned, image-correct, text-correct, and both-wrong (A/V/T/N) evidence states as one bundle. Its objective couples condition-wise correctness with transition rewards for answer invariance, source equivariance, and answer-to-abstention switching, while a primal-dual controller balances unsafe answers against excessive deferral. We also contribute XMC (eXtended Modal Conflict), a four-condition conflict training resource, and evaluate transfer on CMC-Bench and Modality-Bias. Across multiple seeds, CARGO-VL improves conflict handling, unsupported-answer avoidance, and modality balance over pointwise baselines. Ablations identify complementary benefits from relational transition signals and adaptive risk control, supporting counterfactual consistency as a practical objective for reliable multimodal evidence arbitration.
Summary
Vision-language models often receive both images and retrieved text that can conflict or jointly fail to support a reliable answer. Current post-training methods evaluate instances in isolation and therefore fail to enforce consistent behavior when evidence is altered in controlled ways. CARGO-VL addresses this by treating matched counterfactual bundles—covering aligned, image-correct, text-correct, and both-wrong evidence states—as a single optimization unit rather than independent prompts.
The framework couples per-condition accuracy with transition rewards that reward answer invariance under aligned evidence, source equivariance when one modality is made correct, and a switch to abstention when both sources are invalid. A primal-dual controller maintains explicit budgets on unsafe answers and excessive deferral, while a soft minimum protects performance on the weakest condition within each bundle. The method builds on group-relative policy optimization but redefines the group around these counterfactual evidence variants.
To support training without test leakage, the authors release XMC, a four-condition conflict dataset derived from fresh TextVQA and ScienceQA items. Experiments on CMC-Bench and Modality-Bias show consistent gains in conflict handling, unsupported-answer avoidance, and modality balance compared with pointwise baselines, with ablations confirming the value of both the relational transition signals and the adaptive risk constraints.
Why it matters
Offers novel technical depth on trustworthy multimodal AI with strong novelty, reproducibility elements, and alignment to EU ethical AI priorities; actionable for Dutch researchers and advanced labs working on VLMs.
