Securing Multimodal AI through Internal Information Decomposition
06:00 · July 27, 2026 · arXiv cs.AI RSS
Multimodal large language models introduce attack surfaces absent in unimodal systems: adversaries can distribute malicious intent across modalities to evade unimodal safeguards. This motivates using cross-modal consistency as a detection signal rather than inspecting each modality in isolation. Our key observation is that benign inputs induce compatible predictive behavior from text-only and vision-only reasoning that stabilizes when fused, whereas adversarial manipulation disrupts this consistency, causing abnormal multimodal behavior. Existing defenses that examine raw inputs or outputs overlook this internal fusion process, rendering them brittle and computationally expensive. We propose FlowGuard, a lightweight inference-time framework that detects harmful inputs by monitoring internal multimodal consistency. Unlike approaches that rely on scalar confidence metrics, FlowGuard derives FlowVectors inspired by Partial Information Decomposition that quantify cross-modal redundancy, synergy, and modality-specific dominance, capturing whether fused multimodal predictions remain aligned with unimodal semantic evidence. In a one-class classification problem trained solely on benign data, FlowGuard reduces Attack Success Rates from >90% to <15% on unseen attacks, with <3% utility loss and up to a 6 times latency reduction. Our results demonstrate that monitoring cross-modal consistency offers an efficient and effective defense for multimodal reasoning.
Summary
Multimodal large language models combine visual and textual signals during inference, creating attack surfaces that allow adversaries to split malicious intent across modalities so that neither appears harmful in isolation. FlowGuard counters this by probing an MLLM under three separate conditions—vision-only, text-only, and joint multimodal—and comparing the resulting first-token predictive distributions.
The method derives compact FlowVectors from Partial Information Decomposition concepts to measure cross-modal redundancy, synergy, and modality-specific dominance. These features reveal whether the fused multimodal output remains aligned with unimodal semantic evidence or exhibits the misalignment that typically accompanies successful jailbreaks. Because the detector is trained solely on benign data as a one-class Isolation Forest, it requires no adversarial examples and operates without modifying the underlying model.
In experiments across multiple MLLM architectures and multimodal jailbreak benchmarks, the approach reduced attack success rates on unseen threats from above 90 percent to below 15 percent. Benign utility declined by less than 3 percent while inference latency dropped by as much as a factor of six relative to diffusion-based verification techniques. The design therefore supplies a lightweight, process-level signal that targets fusion anomalies rather than surface-level input or output properties.
Why it matters
This research is highly relevant for Dutch AI researchers and practitioners focusing on AI safety and compliance with the EU AI Act. It provides a novel, actionable, and computationally efficient method to secure multimodal AI systems against sophisticated adversarial attacks, aligning with the Netherlands' strategic emphasis on robust and ethical AI deployment.
