LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs
06:00 · August 7, 2026 · arXiv cs.AI RSS
Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals, providing limited evaluation of cross-domain behavioral personalization, where responses must be grounded in heterogeneous daily-life activities. To address this gap, we introduce LUNAR, the first benchmark for evaluating how LLMs personalize responses from longitudinal app interaction histories across universal daily-life domains, including clothing, food, housing, and mobility. To support scalable benchmark construction while mitigating data sparsity and privacy concerns, LUNAR uses a multi-stage coarse-to-fine synthesis pipeline grounded in real-world behavioral patterns. Fidelity analyses show closer alignment with real behavioral distributions than other synthetic benchmarks. Experiments on 19 mainstream LLMs show that access to behavioral logs is necessary but not sufficient for deep personalization: neither more context nor larger models guarantees better performance; effective personalization depends on selecting and integrating relevant evidence across domains. Direct retrieval of fine-grained behavioral records consistently outperforms compressed memory, while stronger personalization can come at the cost of privacy protection. These findings identify evidence selection, cross-domain integration, and privacy control as key challenges for personalized LLMs.
Summary
LUNAR addresses a gap in existing personalization benchmarks, which typically rely on textual personas or single-domain behavioral traces and therefore offer limited insight into how models should ground responses in heterogeneous, longitudinal user activity. The new benchmark evaluates LLMs on their ability to generate personalized answers from app-interaction histories that span four daily-life domains: clothing, food, housing, and mobility. Each query may require evidence drawn from multiple domains, forcing models to identify relevant records while discarding irrelevant ones.
To create a large-scale test set without exposing real user data, LUNAR employs a multi-stage coarse-to-fine synthesis pipeline anchored in anonymized real-world behavioral distributions. Fidelity checks confirm that the resulting synthetic logs align more closely with observed patterns than those produced by prior synthetic benchmarks. The construction process yields both the behavioral histories and the ground-truth evidence sets needed for automatic evaluation.
Experiments covering 19 mainstream LLMs reveal that simply providing behavioral logs or increasing model size or context length does not reliably improve personalization. Performance hinges on accurate selection and cross-domain integration of evidence; direct retrieval of fine-grained records consistently outperforms approaches that rely on compressed memory summaries. Stronger personalization also correlates with reduced privacy protection, highlighting a measurable trade-off.
The benchmark therefore isolates three core open problems: reliable evidence selection from noisy histories, integration of signals across disparate domains, and controllable privacy preservation during personalization.
Why it matters
Strong alignment with Dutch/EU priorities on ethical, transparent, and privacy-preserving AI; actionable for testing personalized LLM services in regulated contexts; high technical depth and novelty for researchers.
