Democratizing AI with Small Language Models: Structured Benchmarking and Parameter-Efficient Fine-Tuning for Local Deployment
06:00 · July 21, 2026 · arXiv cs.AI RSS
AI democratization is not primarily a question of matching frontier-scale generality; it is a question of whether capable models can be selected, audited, and specialized under hardware and governance constraints that ordinary institutions can actually satisfy. This paper studies that problem through a controlled evaluation of nine open-weight language models between 135M and 3B parameters on a 1,085-example, 16-topic multiple-choice benchmark designed for structured local deployment. The benchmark emphasizes symbolic precision, constrained formatting, extraction, and short-horizon semantic decision making under a strict one-letter output protocol. A shared parameter-efficient fine-tuning pipeline then adapts a subset of models using 4-bit NF4 quantization with DoRA/LoRA-style adapters on an NVIDIA L4-class budget. In base evaluation, Qwen Coder 3B leads at 75.67% strict accuracy, followed by Qwen2.5 1.5B at 67.10%, Qwen3.5 2B at 64.98%, and Granite 3.3 2B at 64.61%. On the shared 108-example held-out fine-tuning split, adaptation improves Qwen Coder 3B by +26.85 points, SmolLM2 1.7B by +25.92, Qwen2.5 1.5B by +19.44, SmolLM2 360M by +10.18, and SmolLM2 135M by +5.55. Across ranking, topic-level heterogeneity, difficulty strata, failure composition, efficiency frontiers, and topic-conditioned transfer, the same conclusion recurs: a disciplined workflow of benchmark construction, cross-model evaluation, and low-cost specialization already makes a subset of sub-3B models viable as local experts for structured niche workloads.
Summary
The paper reframes AI democratization around the practical question of whether institutions without hyperscale resources can select, evaluate, and specialize compact models for well-defined workloads. Rather than pursuing open-ended generality, it targets structured deployment scenarios in which models must produce short, schema-constrained outputs under a strict one-letter protocol. Nine open-weight models ranging from 135M to 3B parameters are assessed on a purpose-built 1,085-example multiple-choice benchmark spanning sixteen topics that emphasize symbolic precision, extraction, rule following, and short-horizon decision making.
Base evaluation reveals clear performance differences under identical prompting and scoring conditions. Qwen Coder 3B achieves the highest strict accuracy at 75.67 percent, followed by Qwen2.5 1.5B at 67.10 percent and Qwen3.5 2B at 64.98 percent. Topic-level and difficulty-stratified breakdowns expose substantial heterogeneity, showing that aggregate scores alone mask important variation in failure modes and prior-label bias across binary, ternary, and four-way choice sets.
A shared parameter-efficient fine-tuning pipeline then adapts selected models using 4-bit NF4 quantization together with DoRA- and LoRA-style adapters on NVIDIA L4-class hardware. On a 108-example held-out split, adaptation yields substantial gains: Qwen Coder 3B improves by 26.85 points, SmolLM2 1.7B by 25.92 points, and Qwen2.5 1.5B by 19.44 points. Even the smallest models register measurable lifts, indicating that low-cost specialization can convert general compact checkpoints into viable local experts for narrowly scoped, format-sensitive tasks.
The study underscores that reproducibility, deterministic output protocols, and deployment-oriented metrics such as efficiency frontiers and topic-conditioned transfer are essential for responsible local use. Its central contribution is therefore methodological: a repeatable workflow of benchmark construction, cross-model comparison, and targeted adaptation already renders a subset of sub-3B models practical for constrained institutional workloads.
Why it matters
Directly actionable for Dutch SMEs and institutions seeking ethical, local, auditable AI under EU constraints; provides concrete workflows, benchmarks, and PEFT recipes that match Netherlands priorities on transparency and accessible infrastructure.
