Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices
06:00 · July 27, 2026 · arXiv cs.AI RSS
Accurate latency prediction is critical for deploying large language models (LLMs) on heterogeneous edge devices, where inference latency is affected by model architecture, prompt behavior, runtime backend, hardware utilization, dynamic voltage and frequency scaling (DVFS), and thermal variation. This paper presents a runtime-aware latency prediction framework for deployment-oriented LLM selection. The framework represents each inference request as a hardware-runtime-model-prompt configuration, separates inference into prefill and decode phases, and adaptively fuses static descriptors with dynamic hardware telemetry through a gated prediction model. We evaluate the framework using Pixel mobile devices and validate the profiling pipeline on Jetson Nano, Orange Pi 5 Pro, and an RTX 3090-class GPU platform. On Pixel 8, the full predictor improves total-latency R-squared from 0.953 to 0.960 and decode-latency R-squared from 0.957 to 0.973 over a static-only baseline. On Pixel 8 Pro, it improves prefill-latency R-squared from -1.383 to 0.966. For cross-device transfer, calibration improves Pixel 8 Pro to Pixel 8 total-latency R-squared from -0.974 to 0.940 and decode-latency R-squared from -1.085 to 0.927. Heterogeneous profiling further shows that latency is highly device- and runtime-dependent: the same SmolLM2 model family reaches 8.42 tokens/s on Orange Pi 5 Pro but 64.38 tokens/s on an RTX 3090-class GPU. These results demonstrate that runtime-aware prediction with lightweight calibration can reduce profiling cost and support latency-aware LLM deployment across heterogeneous edge platforms.
Summary
Accurate latency prediction remains essential for selecting large language models suitable for heterogeneous edge hardware, where inference speed varies with model architecture, prompt characteristics, runtime backend, hardware utilization, dynamic voltage and frequency scaling, and thermal throttling. The presented framework addresses this by modeling each inference request as a combined hardware-runtime-model-prompt configuration. It separates the prefill phase, which processes the input prompt and builds the key-value cache, from the decode phase, which generates tokens autoregressively, and employs a gated fusion module to integrate static descriptors of devices and models with dynamic telemetry collected at runtime.
Evaluations on Pixel mobile devices show measurable gains over static-only baselines. On the Pixel 8 the full predictor raises total-latency R² from 0.953 to 0.960 and decode-latency R² from 0.957 to 0.973; on the Pixel 8 Pro it lifts prefill-latency R² from −1.383 to 0.966. Lightweight calibration further enables cross-device transfer, improving Pixel 8 Pro to Pixel 8 total-latency R² from −0.974 to 0.940 and decode-latency R² from −1.085 to 0.927. Profiling across additional platforms confirms that performance is strongly device- and runtime-dependent: the same SmolLM2 model family achieves 8.42 tokens per second on an Orange Pi 5 Pro yet reaches 64.38 tokens per second on an RTX 3090-class GPU.
By supplying calibrated latency estimates before exhaustive device-side execution, the approach supports efficient candidate screening and subsequent Pareto analysis that balances predicted responsiveness against model quality. The resulting workflow reduces profiling overhead while remaining applicable across mobile, embedded GPU, and single-board edge platforms.
Why it matters
Directly actionable for Dutch researchers and SMEs developing on-device LLMs; addresses EU-relevant privacy and edge constraints with novel, reproducible technical methods that advance AI systems research.
