Agentic RAG-VLM: Affordance-Aware Retrieval-Augmented Generation with Self-Reflective Planning for Robotic Grasping
06:00 · July 1, 2026 · arXiv cs.AI RSS
Generalizable robotic grasping in cluttered environments is essential for deploying manipulators in unstructured human spaces, yet existing VLM-based methods rely on visual similarity for object matching, neglecting physical affordances such as handle graspability and material fragility, and operate open-loop without spatial reasoning or failure recovery, limiting their effectiveness when objects are densely packed or physically diverse. We present Agentic RAG-VLM, a unified framework that bridges VLM-based semantic understanding and physically grounded grasp execution by integrating retrieval-augmented generation (RAG) with vision-language models (VLMs) and agentic self-reflective planning. Agentic RAG-VLM introduces three tightly coupled components: (1) a Hierarchical Affordance-Aware RAG (HAA-RAG) that encodes four-dimensional affordance descriptors, including type, material, fragility, and graspable region, and retrieves strategies by functional affordance compatibility rather than visual appearance; (2) a Scene Graph Constraint Reasoner that constructs spatial relationship graphs from VLM perception and translates proximity, occlusion, and support constraints into concrete grasp parameter adjustments; and (3) an Agentic Self-Reflective Pipeline with a 14-type failure taxonomy and three-level adaptive retry for closed-loop grasp refinement. Evaluated on a 12-task benchmark spanning single-grasp, interactive, and long-horizon scenarios with 360 trials per configuration, Agentic RAG-VLM achieves 78.3 percent overall success, a 53.3 percentage-point absolute gain over VLM-only baselines, demonstrating that affordance-aware retrieval, scene graph reasoning, and agentic recovery are jointly essential for robust manipulation.
Summary
Existing VLM-based approaches to robotic grasping often match objects by visual similarity alone, which fails to capture physical properties such as material fragility or graspable regions, and they typically execute plans in open-loop fashion without spatial reasoning or recovery from errors. This limits performance in cluttered scenes where objects vary in density, support relations, and manipulation requirements.
Agentic RAG-VLM addresses these limitations by combining retrieval-augmented generation with vision-language models and agentic self-reflective planning. Its Hierarchical Affordance-Aware RAG encodes objects along four dimensions—type, material, fragility, and graspable region—and retrieves prior grasp strategies according to functional compatibility rather than appearance. A Scene Graph Constraint Reasoner builds spatial relationship graphs from VLM perception and converts proximity, occlusion, and support constraints into adjusted grasp parameters. An Agentic Self-Reflective Pipeline applies a 14-type failure taxonomy together with three-level adaptive retry, enabling closed-loop refinement that maps observed failures to concrete, physics-grounded corrections.
Evaluated across a 12-task benchmark covering single-grasp, interactive, and long-horizon scenarios, the system records a 78.3 percent success rate over 360 trials per configuration. This represents a 53.3 percentage-point improvement over VLM-only baselines, indicating that affordance-aware retrieval, scene-graph reasoning, and structured recovery together provide the necessary grounding for reliable manipulation in unstructured environments.
Why it matters
This research is highly relevant for Dutch AI and robotics researchers, particularly those at technical universities and high-tech industries focusing on automation, logistics, and agri-food. The integration of VLMs with physical affordance reasoning offers actionable, cutting-edge methodologies for improving robotic manipulation in unstructured environments.
