Memory Reward Inflation in Self-Improving LLM Agents
06:00 · August 4, 2026 · arXiv cs.AI RSS
Self-improving LLM agents increasingly learn from experience without updating any weights. Each episode is stored in an external memory, scored, and retrieved for similar future tasks to shape later behavior. Viewed through a reward lens, the stored score is a proxy reward for an implicit, non-parametric policy. Each retrieved episode then becomes a policy-improvement step whose reliability hinges on how that score is produced. In deployment, ground-truth labels are unavailable, so the stored reward is at best an LLM assessment. This substitution creates a failure mode, the *Echo Gap*, across the memory-based self-improving agents and model families studied. Incorrect episodes receive inflated rewards; thus, the agent preferentially reuses the very mistakes it has most confident in. Because the error compounds through memory rather than averaging out and the confirming judge's errors remain correlated with the original self-grading bias, so it cannot identify which memories are overvalued. The missing property is formalized as the *Error-Independence Assumption* (EIA), which we prove is a *necessary* condition for correcting the inflation, not merely a description of a good verifier: a usable signal must track truth *and* decorrelate its error from the memory bias, and the recoverable payoff is a closed-form function of exactly those two quantities. We further show the inflation compounds not only when retrieval ranks by the stored score but also under plain similarity retrieval which is the regime the deployed agent uses. Finally, the answer-free de-inflation algorithm LUCID delivers a consistent end-to-end gain on the BIRD text-to-SQL benchmark. It raises execution accuracy to $56.9\%$, above both a Memento-style self-graded agent ($54.0\%$, a $+2.9$-point mean gain across seeds) and a memory-less agent of identical architecture ($52.4\%$).
Summary
Self-improving LLM agents store completed episodes in an external memory, assign each a utility score, and retrieve relevant entries to guide behavior on new but similar tasks. Because the model parameters remain fixed, performance gains depend entirely on the quality of the stored scores, which function as proxy rewards for an implicit, non-parametric policy. In the absence of ground-truth labels at write time, these scores are typically produced by an LLM grader, creating a closed loop in which the same model family both generates and evaluates its own outputs.
This substitution produces a persistent failure mode the authors term the Echo Gap. Incorrect episodes receive inflated self-assigned rewards; because retrieval favors high-scoring memories, the agent preferentially reuses and reinforces its own mistakes. The inflation does not average out across episodes. Instead, it compounds through repeated retrieval, even when the agent employs plain similarity-based lookup rather than explicit score ranking. The confirming judge’s errors remain correlated with the original grading bias, so additional parametric verification alone cannot isolate overvalued memories.
To diagnose the problem, the authors introduce the Error-Independence Assumption (EIA). A usable corrective signal must both track true task success and produce errors that are decorrelated from the original memory bias. They show that the recoverable improvement is a closed-form function of these two quantities and that EIA is a necessary condition for any de-inflation procedure to succeed.
Building on this analysis, the paper presents LUCID, an answer-free algorithm that applies per-memory utility correction without access to ground-truth labels. When integrated into a Memento-style retrieve-inject-write loop and evaluated on the BIRD text-to-SQL benchmark, LUCID raises execution accuracy to 56.9 percent. This exceeds both the self-graded baseline at 54.0 percent and an otherwise identical memory-less agent at 52.4 percent, confirming that targeted de-inflation can mitigate the Echo Gap under realistic deployment conditions.
Why it matters
This research is highly relevant for Dutch AI researchers and developers building autonomous agents, as it addresses critical reliability and hallucination-reinforcement issues. It aligns strongly with the EU's focus on trustworthy and transparent AI by providing a mathematically grounded method to prevent self-improving models from compounding their own errors.
