Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning
06:00 · June 24, 2026 · arXiv cs.AI RSS
Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason. This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills, limiting generalization to novel problems. We propose Strategy-Guided Policy Optimization (SGPO), which replaces instance-level trajectory imitation with reusable strategy distillation. SGPO extracts structured strategy descriptions from strong-model responses and, for each problem, constructs both autonomous and strategy-guided trajectories to enable direct comparison of the model's behavior with and without strategic guidance. The framework then addresses two key questions. For how to distill, a token-level forward-KL objective selectively transfers the distributional shift induced by strategy conditioning into the unguided policy, with proximal constraints ensuring stability. For when to distill, adaptive instance-level weighting strengthens guidance when autonomous exploration falls short and reduces it as the model's own competence grows. Experiments on four mathematical benchmarks across two model families show that SGPO consistently outperforms SFT, on-policy RL, and hybrid-policy baselines, improving the average score by 2.2 points over the strongest baseline on Qwen2.5-7B-Instruct. Analysis reveals that the forward-KL objective provides an inherently selective distillation signal that outperforms direct trajectory imitation, and that strategy distillation exhibits complementary scaling with base model capability.
Summary
Distilling reasoning from stronger to weaker language models has relied chiefly on supervised imitation of complete solution trajectories. This approach transfers concrete sequences of steps for individual problems rather than the underlying problem-solving logic, which encourages memorization and restricts performance on unseen instances. Strategy-Guided Policy Optimization (SGPO) addresses this limitation by shifting the distillation target from instance-specific traces to reusable strategy descriptions that capture problem type, high-level approach, and general procedural steps without revealing intermediate computations or final answers.
For each training problem, SGPO generates both autonomous trajectories from the student policy and trajectories conditioned on the extracted strategy. It then applies a token-level forward Kullback-Leibler objective to measure and selectively transfer the distributional shift that strategy conditioning produces in the student’s next-token predictions. Proximal constraints at token and trajectory levels stabilize updates, while an adaptive per-instance weighting mechanism increases distillation strength when autonomous performance lags and reduces it as the model’s own competence improves. The method never directly copies any trajectory, whether guided or unguided, preserving reasoning diversity acquired through exploration.
Evaluations across four mathematical reasoning benchmarks and two model families show consistent gains over supervised fine-tuning, on-policy reinforcement learning, and recent hybrid-policy baselines. On Qwen2.5-7B-Instruct the approach raises average performance by 2.2 points relative to the strongest baseline. Further analysis indicates that the forward-KL signal concentrates optimization pressure on tokens whose probabilities change most under strategy guidance, corresponding to critical decision points, and that the benefit of strategy distillation scales complementarily with the base model’s existing reasoning capability.
Why it matters
This research provides advanced methodologies for LLM distillation, which is crucial for Dutch AI researchers aiming to develop efficient, high-performing local models. The shift from memorization to strategy acquisition aligns with the Netherlands' focus on robust, generalizable, and sustainable AI systems.
