AnovaX: A Local, Multi-Agent Voice Assistant with LLM Planning, Typed Executors, and Adaptive Recovery
06:00 · July 20, 2026 · arXiv cs.AI RSS
Desktop voice assistants are still dominated by cloud pipelines that ship raw audio off the machine and expose a fixed set of skills. We describe AnovaX, a small local-first assistant that runs entirely on the user's computer and treats the desktop itself as its action surface. A single Python process wires together a wake-word gate, a speech pipeline, an LLM planner (Gemini) that emits a JSON plan of tool calls, a whitelist-and-denylist safety layer, a multi-agent orchestrator that translates each plan into typed child agents on a bounded thread pool, and an adaptive recovery loop that takes over whenever a core step fails. Every tool corresponds to a specialized agent class (AppAgent, TypingAgent, BrowserAgent and six others) with its own timeout, retry policy, and shared-resource locks. A recursive MetaAgent lets the planner delegate a sub-goal back to itself, capped at two levels of nesting. The recovery loop uses a compact ReAct-style prompt and hides Gemini's latency behind speculative execution of read-only tools. A companion Flask server exposes a phone-friendly remote over the local WiFi, mirrors every agent lifecycle event to the phone in real time, and streams the laptop's screen back over MJPEG so the user can watch remote commands land as they run. The point of the project is less to compete with Siri or Alexa than to show that a legible, few-thousand-line assistant is enough to open apps, type into them, run searches, coordinate concurrent actions, recover from single-step failures, and be driven entirely from a phone in another room -- without the LLM ever touching the keyboard.
Summary
AnovaX is a compact, entirely local desktop voice assistant built around a single Python process that coordinates wake-word detection, speech recognition, and an LLM-based planner. The planner, currently Gemini, produces a structured JSON plan of tool calls; this plan is then handed to a multi-agent orchestrator rather than executed directly. Each tool in the schema maps to a dedicated, typed child-agent class—such as AppAgent, TypingAgent or BrowserAgent—that carries its own timeout, retry policy and resource locks. Up to eight such agents can run concurrently on a bounded thread pool, allowing independent actions to proceed in parallel while a parent process retains control of scheduling and termination.
A two-stage safety mechanism inspects every plan, including those generated recursively by the MetaAgent delegation tool, before any child agent is instantiated. The first stage applies prompt-level constraints; the second enforces an explicit Python whitelist and denylist. When a core step fails, control passes to an adaptive recovery loop that issues a compact ReAct-style prompt to the LLM for the next batch of actions. To mask planner latency, the loop speculatively executes read-only tools from the remaining plan in the background and merges their results once the next batch is approved. Both the static and recovery paths share the same orchestrator and safety filter, keeping the overall codebase to roughly 1,800 lines.
A lightweight Flask server exposes a phone-friendly remote interface over the local network. The server streams lifecycle events to the mobile client via server-sent events and mirrors the laptop screen over MJPEG, allowing the user to observe command execution from another room. All audio processing and planning remain on the host machine; the only external call is the optional Gemini request, which can be disabled in favour of a simple regex fallback. The design deliberately trades the breadth of commercial cloud assistants for inspectability: every proposed action is visible, auditable and revocable before it touches the desktop.
Why it matters
This research is highly relevant to the Dutch AI market due to its strong alignment with EU priorities on privacy, transparency, and ethical AI. The local-first, inspectable multi-agent architecture provides a practical blueprint for Dutch researchers and enterprises building GDPR-compliant, autonomous desktop assistants.
