ProcAgent: An Agentic Framework for Procedural Task Guidance on Edge with Human-in-the-Loop
06:00 · July 29, 2026 · arXiv cs.AI RSS
Procedural tasks such as furniture assembly and home repair impose substantial cognitive demands because users must interpret instructions, track task progress, reason about spatial state, and recover from errors while performing physical actions. Prior multimodal assistants have shown promise for procedural guidance, but most rely on cloud inference and fixed always-on perception, making them poorly suited to privacy-sensitive, latency-critical domestic settings. We present ProcAgent, a fully on-device, agentic, vision-based procedural assistant for real-time adaptive guidances on a single NVIDIA Jetson AGX Orin. ProcAgent uses a propose-and-verify architecture that combines low-latency continuous perception, a symbolic task graph, on-demand vision-language verification, and an LLM-based interaction agent. The system continuously proposes user progress, invokes expensive visual reasoning only when ambiguity or likely deviation arises, and supports both reactive question answering and proactive intervention with human-in-the- loop confirmation. We evaluate ProcAgent along four dimensions: perception accuracy, reasoning, task-level performance, and user experience. Despite running entirely on-device, the system maintains responsive interaction, resolving text-only queries in approximately 2 seconds and visually grounded queries in approximately 8 seconds. In a user study with 10 participants completing assembly tasks, ProcAgent receives positive ratings for comprehensibility, actionability, and privacy comfort. These results show that adaptive procedural assistance can be achieved entirely on edge hardware without sacrificing usability.
Summary
ProcAgent addresses the cognitive load of procedural tasks such as furniture assembly by providing adaptive, vision-based guidance that runs entirely on a single NVIDIA Jetson AGX Orin. Unlike prior multimodal assistants that depend on cloud inference or continuous high-cost perception, the system keeps all processing local to preserve privacy and meet the low-latency requirements of domestic environments. It employs a propose-and-verify architecture in which a lightweight perception module continuously monitors the workspace and generates tentative progress hypotheses. These proposals are checked against an explicit symbolic task graph that encodes step ordering and dependencies, triggering an on-demand vision-language model only when ambiguity or likely deviation is detected.
An LLM-based interaction agent manages both reactive responses to user queries and proactive interventions. In reactive mode the agent supplies step guidance or visual verification on request; in proactive mode it monitors progress and intervenes with brief confirmation questions when it suspects an error that could propagate. Human-in-the-loop confirmation is built in so that the system defers to the user whenever perception and task structure disagree, avoiding unnecessary interruptions. This selective invocation of heavier visual reasoning keeps average response times near two seconds for text-only queries and eight seconds for visually grounded ones, while avoiding the thermal and memory limits that would arise from sustained VLM operation on edge hardware.
Evaluation covers perception accuracy, reasoning quality, task-level performance, and user experience. In a study with ten participants assembling furniture, ProcAgent received positive ratings for comprehensibility, actionability, and perceived privacy, with participants noting the benefit of fully on-device operation. The results indicate that the combination of lightweight continuous perception, symbolic state tracking, selective verification, and human oversight can deliver responsive procedural assistance under the joint constraints of privacy, latency, and limited edge compute.
Why it matters
Directly actionable for Dutch researchers and SMEs building privacy-preserving edge AI systems; aligns with EU GDPR/ethical AI priorities and hardware infrastructure focus; high technical depth and novelty in agentic on-device design.
