Same Cluster, 33 Points More Utilization: What Changed Was the Order
21:46 · August 17, 2026 · Hugging Face Blog
Summary
A constraint-aware GPU allocator was benchmarked against a conventional FIFO scheduler on identical hardware running mixed workloads that include training jobs, real-time inference, batch inference, and quantization. The allocator models the scheduling problem as a single grid of GPU assignments across a planning horizon, subject to five explicit constraints that enforce contiguous blocks for batch-style work, limits on timestep-to-timestep churn for elastic services, and non-preemption of running jobs. Real-time inference is represented as a varying demand curve rather than a static daily reservation, while batch jobs are scored by a priority-weighted, time-decaying objective that also penalizes shortfalls in real-time service.
Under contention the FIFO baseline reserves the daily peak demand for each real-time application for the entire day and then places remaining jobs in arrival order. This leaves substantial idle capacity during off-peak hours and commits GPUs to placements that later high-priority or shape-constrained jobs cannot use. The allocator reclaims those troughs for batch work and reorders placements by priority across the full horizon, producing utilization gains of up to 33 percentage points and priority-weighted output improvements between 24.6 % and 105.1 % across five contended scenarios. In one training-heavy case utilization rose from 53.6 % to 87.0 % while value more than doubled.
Even when all jobs share identical priority the allocator still improves utilization and output by planning placements globally rather than sequentially. The system runs a fast heuristic on every request to stay within API latency budgets and can optionally refine the result with the formal model during periodic review. Both approaches depend on workload-specific estimators, because training, inference, and quantization jobs differ sharply in how model size, strategy, and technique translate into GPU-hour demand.
Why it matters
Directly addresses production GPU orchestration challenges (contention, reservations, contiguous blocks, churn) with quantitative benchmarks and implementation details relevant to ML engineers running mixed training/inference workloads on shared hardware.
