A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing
06:00 · August 17, 2026 · arXiv cs.AI RSS
Large Language Model (LLM) serving has become a critical cloud workload, and realistic traces are essential for motivating and benchmarking serving systems. However, existing LLM serving workload studies remain limited in scale and scope. They often observe short time periods and provide limited visibility into how users interact with models in production. As a result, they do not fully capture how LLM serving workloads evolve over time or how user-model interactions shape production traffic. In this work, we further the understanding of real-world LLM serving workloads through both a global characterization and a longitudinal study of a one-year production trace from Chutes. Unlike prior studies, our trace captures full production behavior across many models and users, including both popular and long-tail models. We analyze the workload from aggregate, temporal, model-level, and user-level perspectives, revealing workload evolution and user-model structure that are typically hidden behind aggregate views. To support future research, we will release the full one-year trace with the paper, enabling downstream studies of production behavior without relying on sampled or synthetically generated workloads.
Summary
Large Language Model serving has emerged as a major cloud workload, yet realistic, long-term production traces remain scarce. Most existing studies examine only brief intervals and offer little insight into how individual users interact with different models, leaving gaps in understanding how traffic patterns shift over months and how those shifts affect system design choices such as caching and load balancing.
This paper addresses those limitations through a global characterization and a year-long longitudinal analysis of production traffic collected at Chutes. The trace records complete request streams across dozens of models and thousands of users, encompassing both high-volume popular models and the long tail of infrequently accessed ones. By examining the data at aggregate, temporal, model-level, and user-level granularities, the authors surface workload evolution and user-model interaction structures that remain invisible when only coarse averages are considered.
The study highlights how request rates, model popularity, and user behavior change over time and how these dynamics influence practical serving concerns such as cache effectiveness and load distribution. To enable further research without reliance on synthetic or sampled data, the authors intend to release the full one-year trace alongside the paper.
Why it matters
This research provides a rare, large-scale dataset and analysis of real-world LLM serving workloads, which is crucial for Dutch AI infrastructure researchers and cloud providers aiming to optimize model deployment, caching, and load-balancing. The release of the full trace enables reproducible benchmarking for local AI systems engineering.
