Explicit State Elicitation Is Not Enough: A Controlled Audit of Memory-Policy Classification
06:00 · August 19, 2026 · arXiv cs.AI RSS
Personalized agents must decide whether retrieved user memory should be used, ignored, updated, or queried before it affects a current task. We use this setting to develop an empirical audit protocol for structured intermediate outputs: first audit dataset shortcuts, then isolate bundled prompt changes, check whether intermediate labels are answer-associated, test decomposed semantic evidence, and audit provider-level execution failures. A 480-example synthetic development set initially suggested large gains from a state-structured prompt bundle, but TF-IDF diagnostics showed lexical separability and no positive standalone Ignore cases. We therefore construct a frozen 160-example controlled counterfactual set with 40 matched four-way families and rule-derived reference policies. On this set, exposing the four state definitions improves accuracy, but an isolated explicit state-output field does not significantly improve policy accuracy for Llama-3.3-70B and gives only a marginal, non-significant gain for GPT-OSS-120B. Supplying benchmark-associated state labels shifts policy predictions, but because those labels deterministically map to policies, this is a label-conditioning diagnostic rather than evidence of a faithful internal mechanism. Family-level and seed-stability analyses further show that example-level accuracy overstates counterfactual consistency: complete four-way family success is rare. An exploratory follow-up that elicits decomposed semantic evidence also fails to improve routing for the cleanly evaluated endpoint; the corresponding GPT-OSS condition was unavailable because of provider-side request validation. We evaluate policy classification only, not downstream responses, tool actions, or memory-store mutation.
Summary
Personalized agents must classify retrieved user memory according to one of four policies—Use, Ignore, Update, or Ask—before the memory influences the current task. The authors treat this decision as a structured intermediate step and develop a five-stage empirical audit to test whether forcing models to emit an explicit state label improves downstream policy accuracy. The protocol first checks for dataset shortcuts, then isolates prompt changes, verifies whether supplied labels are answer-associated, tests decomposed semantic evidence, and finally audits provider-level execution failures.
An initial 480-example synthetic development set appeared to show substantial gains from a state-structured prompt bundle on Llama-3.3-70B and Qwen3-32B. TF-IDF diagnostics later revealed strong lexical separability and the complete absence of standalone positive Ignore examples, rendering the set unsuitable for decisive evaluation. The authors therefore froze a controlled 160-example counterfactual set comprising 40 matched four-way scenario families. Within each family the current task text remains byte-identical while the memory records are varied to produce rule-derived reference policies for every label; two blind annotators confirmed agreement with the reference labels on all examples.
On this set, merely exposing the four state definitions raised accuracy, yet adding an isolated explicit state-output field produced no statistically significant improvement for Llama-3.3-70B and only a marginal, non-significant gain for GPT-OSS-120B. Supplying benchmark-associated state labels shifted predictions, but because the states map deterministically to policies this constitutes label conditioning rather than evidence of an internal mechanism. Family-level and seed-stability analyses further showed that example-level accuracy masks low counterfactual consistency: complete success across all four matched variants remains rare. An exploratory attempt to elicit decomposed semantic evidence also failed to improve routing on the cleanly evaluated endpoint.
The study therefore concludes that explicit state elicitation alone does not reliably enhance memory-policy classification and that apparent gains can stem from dataset artifacts or bundled prompt changes. It advocates reusable audit protocols that separate these factors when evaluating structured intermediate outputs in agentic memory systems.
Why it matters
The research provides a rigorous auditing framework for LLM memory mechanisms, which is highly relevant for Dutch AI researchers and enterprises developing trustworthy, personalized AI agents. It aligns with the EU's emphasis on transparent and reliable AI by exposing the limitations of naive prompt engineering.
