BatchDAG: LLM-Planned Execution Graphs for Scalable Ad-Hoc Analysis Over Enterprise Data
06:00 · July 22, 2026 · arXiv cs.AI RSS
Large language models (LLMs) excel at analyzing individual documents but break down on exhaustive, cross-entity analytical questions over enterprise-scale datasets due to context overflow, loss of per-entity attribution, and linear latency from sequential tool calls. We present BatchDAG, a system in which an LLM generates a typed directed acyclic graph (DAG) of operations -- SQL queries, semantic searches, in-memory transforms, parallel fan-outs, and single-shot analyses -- which a deterministic engine evaluates with topological-wave parallelism and structured JSON data flow. A key optimization, entity-aware batching, groups rows by logical entity before fan-out, reducing LLM calls by up to 47x. BatchDAG is not primarily an accuracy improvement over hand-optimized pipelines; rather, it is a general-purpose orchestration layer that replaces multiple hand-engineered workflows with a single system that generates the appropriate execution strategy from natural language. In controlled experiments on 12 transcript-heavy queries, BatchDAG (3.74/5) achieves quality comparable to an expert-designed pipeline (3.25/5) and significantly outperforms a ReAct agent (3.09/5, p<0.01), with superior provenance (77% transcript evidence rate vs. 46-60% for baselines). A controlled ablation shows structured JSON intermediates reduce hallucinations by 27% versus prose summaries (paired t-test, p=0.107, n=12). The planner achieves 98.8% valid-DAG rate across 300 planning calls. In production at Brevian.ai, BatchDAG processes queries over 50,000+ meetings in under 60 seconds, with measured per-query costs of $0.02-$0.24 at published GPT-5.1 pricing.
Summary
BatchDAG addresses the breakdown of conventional tool-augmented LLM agents when handling exhaustive, cross-entity analytical queries over enterprise-scale corpora. Sequential ReAct-style loops suffer from context-window overflow, loss of per-entity attribution, and linear growth in latency as the number of entities increases. The system decomposes such queries into two phases: an LLM planner that emits a typed directed acyclic graph of operations, and a deterministic execution engine that evaluates the graph with topological-wave parallelism.
The DAG is composed from a fixed vocabulary of six step types—sql, search, transform, fan-out, analyze, and compare—four of which incur zero LLM cost at runtime. Steps exchange structured JSON rows rather than prose summaries, enabling database-style joins, filters, and provenance tracking between stages. The sole operation that scales with entity count is fan-out; entity-aware batching groups rows by logical entity before dispatching LLM calls, cutting the number of invocations by as much as 47× relative to naïve row-level batching while preserving complete per-entity context.
Controlled experiments on twelve transcript-heavy queries show that automatically generated BatchDAG pipelines achieve quality scores comparable to expert-designed baselines and statistically outperform ReAct agents, while delivering markedly higher transcript-evidence rates (77 % versus 46–60 %). An ablation further indicates that structured JSON intermediates reduce hallucinations relative to natural-language summaries. In production at Brevian.ai the system routinely answers queries over more than 50 000 meetings in under a minute at per-query costs between $0.02 and $0.24, demonstrating that a single planning-and-execution layer can replace multiple hand-engineered analytical workflows.
Why it matters
This research is highly relevant for Dutch AI researchers and enterprise practitioners as it offers a scalable, cost-effective architecture for processing large datasets with LLMs. Its emphasis on structured data flow and high provenance aligns perfectly with EU requirements for transparent and auditable AI systems.
