The Wiola Architecture for Efficient Small Language Models
06:00 · July 3, 2026 · arXiv cs.AI RSS
We present Wiola, a fully original Small Language Model (SLM) architecture built from first principles, sharing no structural lineage with any existing model family including GPT, LLaMA, Mistral, or Falcon. Wiola introduces five independently novel components: (i) Spiral Rotary Positional Encoding (SRPE), which embeds token positions on a three-dimensional helical manifold combining absolute, relative, and hierarchical positional signals; (ii) Gated Cross-Layer Attention (GCLA), providing each decoder layer with soft cross-attention access to compressed summaries of two preceding layers for inter-layer coherence; (iii) Adaptive Token Merging (ATM), which dynamically merges se mantically redundant adjacent tokens in middle network layers to reduce attention complexity without information loss; (iv) Dual Stream Feed-Forward (DSFF), replacing the conventional MLP with two parallel streams fused by a learned per-dimension gate; and (v) WiolaRMSNorm, a modified normalisation introducing a per-dimension learned offset vector that prevents representation collapse. We provide complete mathematical derivations, architectural block diagrams, complexity analyses, and systematic comparisons against GPT-2, LLaMA-2, and Mistral. Wiola is released in four sizes (120M, 360M, 700M, and 1.5B parameters) and is fully compatible with the HuggingFace Transformers ecosystem, with all 22 architectural unit tests passing.
Summary
Wiola is a decoder-only autoregressive language model constructed entirely from first principles, without any architectural inheritance from the GPT, LLaMA, Mistral or Falcon families. Its design targets efficiency and inter-layer coherence in the small-model regime through five independently derived components. The model is released in four sizes—120 M, 360 M, 700 M and 1.5 B parameters—and integrates directly with the Hugging Face Transformers library, passing all 22 supplied architectural unit tests.
The first component, Spiral Rotary Positional Encoding, places token positions on a three-dimensional helical manifold that simultaneously encodes absolute, relative and hierarchical signals without additional learned parameters. Gated Cross-Layer Attention then supplies each decoder layer with soft cross-attention to compressed summaries of the two preceding layers, improving long-range consistency at negligible extra cost. Adaptive Token Merging operates in the middle third of the network, dynamically collapsing semantically redundant adjacent tokens during training to reduce quadratic attention complexity while restoring exact sequence length at inference.
A Dual-Stream Feed-Forward block replaces the conventional single-stream MLP with two parallel pathways of differing widths and activations, fused by a learned per-dimension gate. Finally, WiolaRMSNorm augments standard root-mean-square normalisation with a per-dimension learned offset vector that shifts the input distribution before normalisation, counteracting the representation collapse observed in deep attention stacks. Together these modifications yield measurable reductions in attention FLOPs and parameter overhead while preserving full compatibility with existing training and inference pipelines.
Why it matters
This research is highly relevant for Dutch AI researchers and SMEs as it provides a novel, efficient, and open-source Small Language Model architecture. SLMs align perfectly with the Netherlands' focus on sustainable, cost-effective, and transparent AI solutions that can be easily deployed by local enterprises without massive compute resources.
