Hands Free, AIs Forward: NVIDIA XR AI Brings Agents to AR Glasses
00:30 · June 17, 2026 · NVIDIA

NVIDIA XR AI is now available in public beta, giving developers a framework for building multimodal AI agents for AR glasses and XR devices.
Summary
NVIDIA has released the public beta of its XR AI framework, a developer library that supports the construction of multimodal AI agents for augmented reality glasses and other extended reality devices. The library connects device inputs with AI models, enterprise data, and accelerated computing to produce agents capable of perceiving surroundings through video, audio, and sensor streams, interpreting spatial context, retrieving information, reasoning about next steps, and executing tasks through software tools.
Building such agents for real-world use requires more than response generation. They must operate with low latency, maintain awareness of fast-changing conditions, and deliver assistance without distracting the user. XR AI addresses these requirements by combining multimodal perception, enterprise retrieval, reasoning models, and agent orchestration, with the NeMo Agent Toolkit handling tool use, workflows, and multi-agent coordination on NVIDIA DGX Spark, DGX Station, and RTX PRO systems.
Early deployments illustrate the framework’s scope. Siemens is testing it for factory maintenance guidance, allowing engineers to query programmable logic controllers and receive contextual instructions through lightweight glasses. AutoBio’s LabOS system applies the same architecture to scientific workflows, guiding researchers through stem cell and gene-editing procedures while recording structured data. Additional pilots at the University of Pittsburgh Medical Center support surgical teams with non-occluding context, while Innoactive and Atlantic Studios explore its use in automotive design reviews and immersive historical experiences.
Why it matters
This update highlights a major advancement in multimodal AI and wearable tech integration. For the Dutch AI ecosystem, it presents new opportunities for developers and SMEs to build innovative XR applications using NVIDIA's infrastructure.








