An LLM-Explainable DRL Framework for Passenger-Directed Autonomous Driving
06:00 · June 23, 2026 · arXiv cs.AI RSS
Autonomous vehicles offer the potential for safer and more efficient mobility, yet public trust remains limited due to the lack of transparency in their decision-making. This work addresses this issue by combining deep reinforcement learning (DRL) for adaptive driving control with large language model (LLM)-based explainability modules designed to communicate agent behavior to passengers. DRL agents were trained in simulation using a Dueling Double Deep Q-Network to follow distinct driving requests: \textit{fast}, \textit{comfort}, and \textit{stop}. They demonstrated stable learning, safe compliance with traffic rules, and reliable switching between modes within a single trip. In parallel, LLM modules were introduced to interpret passenger requests, determine when explanations were needed, and generate concise, safety-oriented justifications. Results show that this framework, serving as a proof of concept for integrating RL decision-making and LLMs, balances safety, adaptability, and explainability, and is most effective when requests are delayed or overridden due to safety constraints.
Summary
A new framework integrates deep reinforcement learning with large language models to make autonomous vehicle decisions more transparent to passengers. The approach pairs a Dueling Double Deep Q-Network agent, trained in the SUMO traffic simulator, with LLM modules that interpret spoken commands and supply real-time explanations. The DRL component learns three distinct longitudinal driving policies—fast, comfort, and stop—on a single-lane urban corridor that includes signalized intersections and an unsignalized pedestrian crossing. Agents demonstrate stable convergence, consistent adherence to traffic rules, and the ability to switch policies within a single trip.
The LLM modules handle three supporting tasks: transcribing passenger speech via Whisper, classifying requests as direct or indirect, and deciding whether an explanation is required. When safety constraints prevent immediate compliance, the system generates concise, passenger-oriented justifications rather than technical traces. This conflict-detection mechanism is triggered most often during overrides, such as halting for a red light or yielding to pedestrians despite a “fast” command.
Evaluated as a proof-of-concept in simulation, the hybrid architecture shows that passenger-directed control and safety-oriented explanations can coexist without sacrificing rule compliance. The design is particularly effective when requests must be delayed or refused, offering a practical route toward greater public trust in autonomous driving systems.
Why it matters
This research aligns with the Dutch AI market's focus on ethical, transparent AI and smart mobility. It provides researchers with a novel approach to Explainable AI (XAI) that could help autonomous systems comply with strict EU transparency regulations.
