Robust Critics: Defending LLMs Against Multi-Turn Attacks
06:00 · July 24, 2026 · arXiv cs.AI RSS
When a user asks a language model something harmful, is it a genuine attack or a misunderstood but well-meaning question? This ambiguity is one of the central challenges of LLM safety. A model that assumes the worst harms legitimate users; one that assumes the best is easily exploited. The problem is compounded in multi-turn dialogue, where an attacker's true intent may only reveal itself gradually across many exchanges, yet existing safety frameworks apply a contextual bandit treatment, ignoring the trajectory of the conversation. To that end, we propose Dialogue Critic Guided Sampling (DCGS), a framework that addresses this by inferring user intent at every turn of dialogue. Instead of applying a fixed rule about what is or is not safe, DCGS learns what the user's intent is likely to be based on the full conversational history and generates responses accordingly. Formally, we model adversarial dialogue as a Markov Decision Process and learn value and regret-based critics at both the individual token and utterance (full response) levels, scoring candidate responses via an action-value critic. We prove that this inference-time reweighting approximates exponential tilting of the base policy, guaranteeing improvement in expected return for any finite candidate pool, a property that group-relative objectives do not exhibit. Evaluated on CARES-18k, WildJailbreak, Redbench, and Harmbench, DCGS outperforms strong robust baselines and frontier models on adversarial dialogue tasks. DCGS also transfers to frontier models, improving their robustness without fine-tuning.
Summary
When a user poses a request that appears harmful, large language models must decide whether the query reflects genuine malice or a benign but poorly phrased intent. Single-turn safety mechanisms struggle with this distinction, and the difficulty grows in multi-turn exchanges where an adversary can gradually steer the conversation through trust-building, context manipulation, and incremental escalation. Existing approaches often treat each turn as an isolated contextual bandit problem, discarding the dialogue trajectory that would allow intent to be inferred over time.
Dialogue Critic Guided Sampling addresses this limitation by modeling the interaction as a Markov Decision Process in which user intent is treated as a latent variable updated at every turn. The method generates candidate responses and then reweights them using learned critics that operate at both the token level and the full-utterance level. These critics incorporate value estimates and regret signals derived from the complete conversation history, enabling the model to distinguish adversarial from benign trajectories without altering the underlying language model parameters.
The reweighting step approximates exponential tilting of the base policy. For any finite set of candidate utterances, the procedure guarantees an improvement in expected return relative to the original policy, a property that does not hold for group-relative objectives. Because the critics are applied only at inference time, the approach remains compatible with frontier models and requires no additional fine-tuning.
Evaluations on CARES-18k, WildJailbreak, Redbench, and Harmbench show that DCGS improves defense success rates against multi-turn attacks while preserving performance on benign queries, outperforming both specialized robust baselines and unmodified frontier systems.
Why it matters
This research is highly relevant for Dutch AI researchers and enterprises focusing on LLM safety and alignment, particularly in light of the EU AI Act's stringent robustness requirements. The proposed inference-time defense mechanism is lightweight and transfers to frontier models, making it highly actionable for local AI deployments.
