ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence
06:00 · August 26, 2026 · arXiv cs.AI RSS
State-of-the-art Natural Language to SQL (NL2SQL) models report execution accuracy exceeding 89 percent on established benchmarks such as Spider and BIRD. However, these benchmarks rely on simplified academic schemas and open-source SQL dialects that do not reflect the complexity of enterprise database environments. We introduce ESQ-Bench, an Oracle-first NL2SQL benchmark with systematic complexity tiers and silent-divergence evaluation across three enterprise schema complexity tiers. We constructed and released six populated schemas (465 tables, 164,682 rows, zero empty tables) with identical seed data on Oracle, PostgreSQL, MySQL, and SQL Server, a four-metric evaluation harness (EM, EX, SR, SD), and 550 gold-validated question-query pairs (Tier-1: 95; Tier-2: 228; Tier-3: 227). Schema-linked prompting with GPT-4o shows monotonic execution-match degradation across tiers: 79.8, 60.3, and 57.2 percent EX on executed queries (June 2026), versus 75.6, 80.4, and 95.8 percent on an earlier 142-question pilot slice. EM stays below 7 percent tier-wide; operational silent-divergence reaches 73 to 99 percent among EX-passing queries. Failure analysis shows wrong-result semantics dominate at higher tiers. Claude Sonnet 4.6 with schema-linked prompts reaches 87.4, 74.9, and 68.7 percent EX (executed queries), exceeding GPT-4o schema-linked on every tier. GPT-4o zero-shot EX on executed queries (78.7, 73.5, and 77.8 percent) inverts schema-linked at Tiers 2 to 3 due to lower execution rates and survivor bias in the zero-shot versus schema-linked analysis. Local Llama 3.2 schema-linked reaches only 13.3 percent bank-wide EX (73 out of 550), underscoring the gap between closed API models and open-weight baselines on enterprise Oracle schemas.
Summary
ESQ-Bench addresses the mismatch between current NL2SQL benchmarks and real enterprise environments by introducing an Oracle-first evaluation suite built on three tiers of schema complexity. Existing benchmarks such as Spider and BIRD rely on small academic schemas averaging roughly five tables, fully documented foreign keys, and SQLite or PostgreSQL dialects. In contrast, enterprise Oracle systems commonly feature hundreds of tables, partial foreign-key enforcement, ambiguous column names, legacy abbreviations, and dialect-specific constructs that existing models rarely encounter.
The benchmark supplies six populated schemas containing 465 tables and 164,682 rows of seeded data, released with identical content across Oracle, PostgreSQL, MySQL, and SQL Server. It includes 550 gold-validated question-query pairs distributed across the tiers (95 / 228 / 227) and a four-metric evaluation harness that measures exact match, execution match, semantic recall, and silent semantic divergence. The latter metric captures cases in which a generated query executes successfully yet returns results that diverge semantically from the intended answer, often due to Oracle-specific NULL handling, empty-string coercion, or ordering defaults.
Schema-linked prompting experiments with GPT-4o reveal clear performance degradation as complexity increases, with execution-match scores falling from 79.8 percent on Tier 1 to 57.2 percent on Tier 3. Exact-match rates remain below 7 percent across all tiers, while operational silent-divergence rates among execution-correct queries range from 73 to 99 percent. Failure analysis indicates that wrong-result semantics become the dominant error category at higher tiers. Comparative runs show Claude Sonnet 4.6 outperforming GPT-4o under the same prompting regime, whereas an open-weight Llama 3.2 baseline reaches only 13.3 percent execution match across the full set.
By releasing the schemas, seed scripts, question bank, evaluation harness, and baseline results, the work enables reproducible assessment of NL2SQL generalization to production Oracle environments and highlights the limitations of metrics that ignore silent semantic differences.
Why it matters
High technical depth and reproducibility make it directly usable by Dutch AI researchers developing or evaluating database interfaces. Novel focus on Oracle dialect and enterprise complexity aligns with real-world deployment challenges faced by advanced practitioners.
