When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents
06:00 · July 22, 2026 · arXiv cs.AI RSS
LLM agents are increasingly used as transaction compilers: a user states an intent in natural language, and the model emits a structured object that an API can execute. JSON Schema and provider-level structured-output modes are useful because they remove a large class of parse failures, but they do not by themselves decide whether the object is a safe, faithful transaction. We introduce OrderBench, a deterministic benchmark for restaurant ordering agents that separates syntactic validity, schema validity, status decisions, exact item semantics, constraint preservation, and unsafe acceptances. Across 2,400 Nebius Token Factory calls to four open models in prompt-only and JSON-schema modes, we find that schema-valid output can still have large semantic error rates. In the strongest model, both modes achieve 100% schema validity, yet semantic success remains near 80%; in weaker models, schema-valid unsafe acceptances occur in double digits. The result is a concrete engineering warning: structured output is a necessary interface layer, not a substitute for domain verification and fail-closed execution.
Summary
LLM agents deployed as transaction compilers translate natural-language requests into structured objects that downstream APIs can execute directly. In domains such as food ordering, the critical failure mode is no longer malformed JSON but well-formed objects that misrepresent item availability, modifier scope, allergen constraints, or dietary rules. OrderBench isolates this semantic layer by providing a deterministic 300-case benchmark built around a seven-SKU menu, hand-coded oracles, and ten edge-case categories that include negated modifiers, scoped modifiers, allergen conflicts, and unavailable catalog entries.
The benchmark records seven orthogonal outcomes for each generated object: JSON validity, schema validity, status correctness (accepted, needs_clarification, rejected_safety), exact multiset equality over items and modifiers, constraint preservation, overall semantic success, and unsafe acceptance. Unsafe acceptance is defined strictly as emitting an accepted status for any order that violates an allergen, dietary, or availability rule. Evaluation was performed at temperature zero on four open models through a single OpenAI-compatible endpoint, comparing a prompt-only baseline against the provider’s strict JSON-schema mode across identical cases and prompts.
Results show that perfect schema compliance does not imply semantic reliability. The strongest model reached 100 % schema validity in both modes yet achieved only 81–83 % semantic success. Weaker models produced 100 % schema-valid outputs while recording semantic success below 40 % and unsafe-acceptance rates above 40 %. Category-level breakdowns indicate that the largest error clusters arise from availability checks and constraint handling rather than generic formatting. The released artifacts include raw responses, verifier code, and per-case oracles, enabling reproducible measurement of the gap between structural validity and executable correctness.
Why it matters
Directly relevant for Dutch AI researchers studying reliable, ethical LLM deployment in enterprise/SME settings; highlights EU-aligned safety needs for agentic systems under potential AI Act constraints.
