SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction
06:00 · July 20, 2026 · arXiv cs.AI RSS
Mobile graphical user interface (GUI) agents have demonstrated remarkable capabilities in automating complex tasks, yet they introduce critical safety risks where a single erroneous action can lead to irreversible consequences. Existing safety mechanisms are primarily reactive, lacking the ability to assess risks before execution. In this paper, we introduce SeerGuard, a consequence-aware safety framework designed to mitigate these risks through pre-execution instruction-level screening and action-level risk assessment. Specifically, the action-level assessment analyzes agent-proposed actions within current GUI states, anticipating likely outcomes to identify risks before they are executed. To enable these capabilities, we construct a unified safety-augmented world model (SAWM) via multi-task learning, integrating semantic next-state prediction with safety risk assessment. Extensive experiments demonstrate that SeerGuard generalizes effectively across diverse mobile GUI agents. On Qwen3-VL-8B-Instruct, it increases the safety-utility score from $0.191$ to $0.596$ at $\omega=0.8$ and reduces the risk-cost score from $0.347$ to $0.130$ at $\alpha=0.8$. Further analyses on our SAWM validate the effectiveness of the instruction-level screening, alongside the capability of action risk assessment and next-state prediction.
Summary
SeerGuard addresses a core limitation in mobile GUI agents powered by vision-language models: their tendency to execute actions whose consequences cannot be reversed once triggered on live device interfaces. Unlike earlier defenses that either scan user instructions for overt malice or review outcomes after execution, the framework adds consequence-aware checks that anticipate state changes before any tap or swipe occurs. It combines coarse instruction-level filtering with finer action-level evaluation, rejecting requests that are explicitly harmful while also flagging context-dependent risks that only become visible once the current screen is taken into account.
The technical backbone is a Safety-Augmented World Model (SAWM) trained through multi-task learning on Qwen3-VL-8B-Instruct. Given a screenshot and a candidate action, the model produces a concise semantic description of the expected next state, then assesses whether that state would be unsafe and supplies a short rationale. Training data are assembled from general textual safety corpora, multimodal mobile risk examples, and synthetically generated mobile scenarios, allowing the model to learn both malicious-instruction detection and forward simulation without requiring pixel-level screen generation. This semantic approach keeps inference lightweight enough for online use while focusing on functional outcomes that matter for safety decisions.
Evaluation on MobileSafetyBench shows consistent gains in the safety-utility trade-off. When SeerGuard is integrated, the safety-utility score on Qwen3-VL-8B-Instruct rises from 0.191 to 0.596 at ω = 0.8, while the risk-cost score falls from 0.347 to 0.130 at α = 0.8. Radar-chart analysis across six task categories further indicates higher refusal rates on high-risk operations without a proportional drop in successful completion of benign tasks. The same pattern holds across several other VLM-based agents, suggesting that the pre-execution world-model check generalizes beyond any single backbone.
Why it matters
This research is highly relevant for Dutch AI researchers and developers focusing on agentic AI and AI safety. It aligns with the EU's stringent regulatory emphasis on safe, transparent, and risk-aware AI systems by offering a proactive mechanism to prevent harmful autonomous actions before they occur.
