SkillHarness: Harnessing Safe Skills for Computer-Use Agents
06:00 · June 23, 2026 · arXiv cs.AI RSS
Computer-Use Agents (CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need for continual skill learning during interaction. Recent approaches address this challenge by learning reusable skills from successful trajectories. However, these skill learning methods largely assume static and safe environments, overlooking risks from adversarial interactions (e.g., prompt injections) and environmental dynamics (e.g., pop-ups). In dynamic settings, such assumptions can lead to risky skill learning and brittle execution, undermining the reliability of CUAs. This raises the question: how can CUAs learn and use skills safely in dynamic environments? To address this problem, we propose SkillHarness, a framework for safe skill harnessing in dynamic environments. SkillHarness moves beyond static skill abstractions by modeling skill learning and utilization as a safety-constrained interaction process. Specifically, we introduce the skill boundary that leverages multi-source supervision signals to identify safe skills from interaction trajectories, and construct self-improving safety constraints throughout the skill lifecycle. In addition, SkillHarness introduces selective skill reuse, where tasks are guided to decompose according to context and completed through the selective activation of skill subsets. Our experiments demonstrate that SkillHarness significantly reduces the unsafe rate of learned skills by 57.1% and consistently improves execution stability under dynamic environmental changes, outperforming existing baselines.
Summary
Computer-Use Agents operating in dynamic interactive environments must continually acquire reusable skills from trajectories, yet existing methods typically extract patterns only from successful outcomes under the assumption of static, benign conditions. This leaves them exposed to supervision bias, in which transient or unsafe states become encoded in learned skills, and to brittle execution when fixed procedural abstractions encounter distribution shifts such as pop-ups or altered UI flows.
SkillHarness reframes skill induction and reuse as an explicitly safety-constrained interaction. It introduces a skill boundary that integrates three complementary signals: positive examples from successful trajectories, lessons from failure cases that reveal non-generalizing behaviors, and risk indicators drawn from adversarial or changing contexts. These signals are used to construct self-improving safety constraints that persist across the skill lifecycle, allowing representations to capture not only executable patterns but also the conditions under which those patterns remain reliable.
At execution time, the framework decouples high-level intent from environment-specific implementation through selective skill reuse. Macro skills encode strategies together with their success patterns and behavioral constraints, while micro skills supply parameterized code grounded in the current state. A planner activates only those skills whose constraints are satisfied and falls back to flexible LLM-based planning when no suitable skill applies.
Across multiple benchmarks, SkillHarness reduces the unsafe rate of learned skills by 57.1 percent compared with prior trajectory-based approaches, improves safety during skill utilization by an average of 31.9 percent, and raises task success rates by 19 percent through more stable execution under environmental change.
Why it matters
This research directly supports the Dutch and EU strategic focus on safe, ethical, and reliable AI deployment. For researchers and advanced practitioners in the Netherlands, it provides actionable methodologies to build autonomous agents that comply with stringent safety constraints in dynamic environments.
