Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
06:00 · July 3, 2026 · arXiv cs.AI RSS
Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal. However, the richer procedural information in the rollout is rarely retained or reused. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur. We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory-free model at inference. PMD organizes the memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories. A memory-conditioned self-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters. The central design principle is co-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy. Empirically, across Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Co-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains.
Summary
Reinforcement learning with verifiable rewards (RLVR) and its self-distillation extension SDPO update a language-model policy from episode-level verification signals. This approach discards the richer procedural content generated across multiple rollouts, such as recurring solution strategies, persistent failure modes, and cross-problem patterns that only become visible when episodes are compared under an evolving policy.
Procedural Memory Distillation (PMD) addresses this gap by continuously extracting three layers of procedural knowledge directly from the model’s own trajectories: raw solution paths, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems. The extracted memory serves as a temporary training scaffold; a memory-conditioned self-teacher then uses the accumulated experience to supervise the student model on its latest rollouts, gradually internalizing the procedural knowledge into the policy weights.
The method is built on an explicit co-evolution loop in which the current policy produces new trajectories that refresh the memory, while the updated memory in turn shapes the supervision signal that refines the policy. At inference the memory is no longer required, leaving a standard model whose parameters already encode the distilled procedures.
Experiments on Qwen3-8B and OLMo3-Instruct-7B show consistent gains over SDPO: 3.8–5.5 % on SCIKNOWEVAL and 7.9–13.6 % on LIVECODEBENCH. Ablations confirm that freezing either the memory or the policy during training erodes most of these improvements, underscoring the necessity of the joint update dynamic.
Why it matters
Novel self-improvement method for LLMs offers actionable technical insights for Dutch AI researchers and advanced practitioners focused on model training and reasoning enhancement.
