Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness
06:00 · August 6, 2026 · arXiv cs.AI RSS
Benchmarking machine unlearning methods is critical to understand whether sensitive knowledge is removed from large language models (LLMs) or not. Current unlearning benchmarks include mainly single-hop questions and a narrow set of multi-hop questions. Although effective, they still face two challenges. (1) Knowledge is not isolated, whereby diverse multi-hop reasoning paths can potentially induce knowledge leakage than normal queries. (2) Unlearning may be fragile: unlearned knowledge can be partially recovered through recovery attacks such as lightweight post-unlearning adaptation, making static evaluation insufficient. Therefore, in this paper, we introduce \unlearning as a novel benchmark to understand robust LLM knowledge removal across diverse reasoning paths and recovery attacks. We experiment with this benchmark on 3 models, 6 unlearning methods, and 2 carefully curated datasets. Results show that existing methods are vulnerable to multi-hop reasoning paths and recovery attacks. We further explore the trade-off among forget quality, robustness, and model utility for LLM unlearning.
Summary
The paper presents Leak-resistant Unlearning, a benchmark designed to test whether machine-unlearning methods truly remove sensitive knowledge from large language models. Existing evaluations rely mainly on single-hop questions or a limited set of chain-style multi-hop queries, which fail to capture how knowledge remains entangled across multiple facts. The new benchmark therefore adds two dimensions: six logic-inspired multi-hop reasoning structures that probe different inference paths, and three recovery attacks that attempt to restore forgotten information through lightweight post-unlearning adaptation or prompt manipulation.
The evaluation framework measures two practical properties. Path consistency checks whether target knowledge stays inaccessible when models are queried through varied reasoning routes rather than direct prompts. Recovery robustness assesses whether the same knowledge can be elicited after attackers apply parameter-level or prompt-based interventions without additional data. A data-construction pipeline extracts structured facts from existing sources, composes them into the six reasoning patterns, and applies automated verification to ensure question quality.
Experiments were run on three models, six unlearning methods, and two curated datasets derived from MQuAKE and Books. Results indicate that current techniques remain vulnerable: certain multi-hop structures leak substantially more information than single-hop or simple chain queries, and unlearned knowledge is frequently recoverable. The study also documents clear trade-offs among forget quality, robustness against attacks, and retained model utility, showing that no existing method simultaneously satisfies all three objectives at a high level. The benchmark and associated datasets are released to support more rigorous assessment of privacy-preserving unlearning.
Why it matters
Strong EU relevance for GDPR-compliant unlearning and ethical AI deployment; actionable benchmark for Dutch researchers and SMEs developing privacy-aware LLMs; high technical depth and reproducibility support advanced analysis.
