APeB: Benchmarking Personalization Ability of Large Language Model Agents
06:00 · July 7, 2026 · arXiv cs.AI RSS
LLM-powered agents struggle with personalization when users issue raw, underspecified queries. In this setting, agents must infer latent intent, extract preferences from noisy interaction histories, and select among competing alternatives. Existing benchmarks rarely test this capability, as they often rely on user-refined queries or simplified histories. We introduce personalized product search (PPS), a testbed for agentic personalization under raw queries and diverse histories. We construct Agent Personalized Benchmark (APeB) from action logs, pairing underspecified intents with rich histories and user-viewed candidate items. Evaluating state-of-the-art LLMs with multi-step agent workflows, we find that models handle explicit queries well but struggle with early-stage queries requiring intent and preference discovery. Rubric analysis attributes this gap mainly to ineffective history use. A simple history-aware query-refinement pipeline, VQRA, yields consistent gains, highlighting the need for dedicated history-utilization modules in personalized agents.
Summary
LLM-powered agents often receive vague, early-stage user queries that omit explicit constraints or preferences. In such cases the agent must infer latent intent, surface relevant signals from noisy, heterogeneous interaction histories, and choose among closely competing options. Existing benchmarks typically sidestep this setting by supplying already-refined queries or simplified histories, leaving the joint challenge of intent discovery and history-grounded preference modeling largely untested.
The authors address the gap by defining personalized product search as a concrete testbed and constructing the Agent Personalized Benchmark (APeB) from real behavioral logs on a large content platform. Each of the 5,648 retained cases pairs an underspecified search intent with up to 60 days of prior activity (videos, livestreams, and product interactions) and a small set of heavily viewed candidate items that culminate in an observed purchase. Sessions are filtered to ensure non-trivial query reformulation and extensive comparison, so that success requires genuine user-specific reasoning rather than surface-level semantic matching.
When state-of-the-art LLMs are embedded in multi-step agent workflows and evaluated on APeB, performance is strong on explicit, late-stage queries or randomly sampled candidates. On early-stage intents paired with hard candidates, however, accuracy falls sharply. Rubric-based analysis attributes the drop primarily to ineffective use of history: models frequently overlook or misweight preference signals and produce weak intent attributions. Adding standard ReAct-style planning yields little or no improvement and can even degrade results through compounding planning errors.
To mitigate the observed weakness, the authors introduce VQRA, a lightweight history-aware query-refinement pipeline that first grounds the raw query in retrieved history before issuing a more precise search. The approach produces consistent gains in intent inference and Hit@1, indicating that dedicated history-utilization components remain an open requirement for practical personalized agents.
Why it matters
This research provides a valuable benchmark and methodology for Dutch AI researchers and enterprises, particularly in e-commerce and customer service, developing personalized LLM agents. Improving intent discovery from user histories directly impacts the effectiveness of AI-driven consumer applications prevalent in the Netherlands.
