COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows
06:00 · July 3, 2026 · arXiv cs.AI RSS
Agents are increasingly used to construct workflows and assist humans in completing recurring tasks more efficiently. As these workflows become repeated and domain-specific, agent memory and reusable skills become increasingly important: agents should be able to recall workflow patterns, execution constraints, and user preferences from previous runs. We study this problem in workflow-based image generation and introduce COMFYCLAW, an agentic skill evolution harness for controlling ComfyUI workflows. COMFYCLAW formulates workflow construction as typed graph editing, exposes tools organized by construction stage, automatically reverts invalid edits, and uses a region-level vision-language model (VLM) verifier to translate visual failures into actionable repair suggestions. The framework further evolves a progressively disclosed skill library, where trajectories, execution errors, and verifier feedback from previous runs are distilled into reusable Agent Skills. Across four benchmark splits, three agent models, and two image backbones, COMFYCLAW achieves the best average image-generation evaluation score across all six agent configurations, outperforming a verifier-only baseline without skill evolution. Human annotations further show that annotators prefer COMFYCLAW over variants without skill evolution. Our results suggest that skill evolution is an effective mechanism for improving agent reliability and performance in recurring visual workflow construction.
Summary
COMFYCLAW is an agentic framework that treats ComfyUI workflow construction as typed graph editing rather than free-form prompt refinement. The system stages tool access by construction phase, automatically reverts edits that would break execution, and supplies the controlling agent with runtime feedback from the unmodified ComfyUI environment. A region-level vision-language model verifier then inspects generated images against the original requirements, converting localized visual failures into concrete repair directives that the agent can apply in subsequent graph edits.
Beyond single-run correction, COMFYCLAW maintains a skill-evolution loop. Execution trajectories, verifier critiques, and successful repairs are distilled into reusable Agent Skills that are validated on held-out tasks before being added to a progressively disclosed library. Later runs can invoke these skills directly, allowing the agent to reuse validated patterns instead of rediscovering them. The library grows only when new skills demonstrably improve performance under a graph-complexity prior, limiting unchecked proliferation.
Across four benchmark splits, three agent backbones, and two image-generation models, the full COMFYCLAW configuration records the highest average image-generation score of the six evaluated setups. It surpasses a verifier-only baseline that lacks skill evolution and a no-refinement control by clear margins. Human raters also prefer outputs produced with evolved skills. In the reported runs, the 318 committed skills account for roughly half of all skill invocations after the initial learning phase, indicating that the mechanism converts repeated experience into stable, reusable control knowledge for recurring visual workflows.
Why it matters
This research is highly relevant for AI researchers and advanced practitioners in the Netherlands focusing on generative AI and autonomous agents. The proposed self-evolving skill framework offers actionable methodologies for Dutch tech SMEs and creative industries looking to optimize and automate complex image generation workflows.
