Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
06:00 · July 14, 2026 · arXiv cs.AI RSS
Prompt wrappers often differ only in formatting, yet they can change model scores enough to flip leaderboard conclusions. We study this variance under a token-controlled protocol and introduce two complementary metrics: the Format Sensitivity Index (FSI), the accuracy range induced by wrapper choice, and the Parseability Sensitivity Index (PSI), the corresponding range in answer parseability. Across 140,000 OpenRouter generations spanning 7 QA tasks, 5 wrapper families, and 4 instruct models from 7B to 72B parameters, we find that mean FSI varies by over 30x across models and is largely explained by compliance failures. A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper. We argue that reporting accuracy without wrapper variance and compliance is statistically fragile, and we give practical recommendations for both benchmarking and structured-output deployments.
Summary
Prompt wrappers—templates that add formatting instructions such as JSON schemas, key-value delimiters, or scratchpad sections around otherwise identical task content—can alter LLM benchmark scores by amounts large enough to reverse model rankings. The study quantifies this effect through two range-based metrics. The Format Sensitivity Index measures the spread in accuracy across five wrapper families, while the Parseability Sensitivity Index tracks the corresponding spread in the fraction of outputs that an automated extractor can map to a canonical answer. Both are evaluated under a token-controlled protocol that pads prompts to a fixed character budget and logs residual token variation, across 140,000 generations on seven question-answering tasks and four instruct models ranging from 7 B to 72 B parameters.
Results reveal extreme model-dependent differences. Mean FSI varies by more than thirtyfold, with Qwen-2.5-72B remaining nearly invariant (mean FSI 0.024) while Phi-4 exhibits a mean FSI of 0.763; normalized sensitivity accentuates the contrast further. A weighted least-squares regression with fixed effects for model, task, and wrapper shows that parseability remains a strong predictor of accuracy (coefficient 0.819) even after these controls, indicating that most wrapper-induced score changes stem from outright compliance failures rather than subtle reasoning effects. Bootstrap confidence intervals and a matched low-token-spread subset corroborate the robustness of these patterns.
The authors conclude that single-point accuracy figures reported without wrapper variance or compliance statistics are statistically fragile. They recommend that benchmarking suites track distributions over wrapper families and that structured-output deployments adopt explicit parseability monitoring or constrained decoding to reduce the observed sensitivity.
Why it matters
Directly actionable for Dutch AI researchers and labs evaluating or deploying LLMs: wrapper variance can flip leaderboard results and affect real structured-output reliability, aligning with EU emphasis on transparent, reproducible AI.
