Woodpecker Distillation: Weak Models Diagnose Reasoning Bugs in Strong Models
06:00 · August 7, 2026 · arXiv cs.AI RSS
Large language models often fail on reasoning tasks despite possessing the capability to solve them. We argue that many such failures arise from localized reasoning bugs in intermediate steps rather than from global incompetence. We show that these bugs are frequently repairable: inserting a short patch generated by a weak probe model after the same strong-model reasoning prefix can redirect the trajectory toward a correct solution. However, this corrective effect is not reliably internalized by directly fine-tuning on weak patches or repaired trajectories, suggesting that the useful signal lies not in the intervention text itself, but in how it reshapes the model's future reasoning distribution. We therefore propose Woodpecker Distillation, a weak-to-strong training framework that learns from contrastive local interventions. Our method contrasts successful and unsuccessful weak-model patches at the same prefix, constructs a corrective teacher distribution from their induced future token predictions, and distills this signal into the strong model. Experiments on mathematical reasoning benchmarks show that Woodpecker Distillation consistently improves strong-model performance and outperforms direct imitation baselines.
Summary
Large language models frequently produce incorrect answers on reasoning problems even when they possess the underlying capability to solve them. The paper attributes many of these failures to localized reasoning bugs—recoverable errors in intermediate steps rather than global incompetence. Experiments show that a short patch generated by a weaker probe model, inserted after an identical prefix produced by the stronger model, can redirect the continuation toward a correct solution on several mathematical reasoning benchmarks.
Direct imitation of the weak-model patches or the resulting repaired trajectories, however, fails to internalize this corrective effect. The authors observe that the value of the intervention lies not in the surface text of the patch but in the way it alters the strong model’s subsequent token distribution. Standard fine-tuning therefore captures little of the useful signal.
Woodpecker Distillation addresses this gap by turning the weak probe into a diagnostic tool. For any given prefix, the method samples multiple patches from the weak model, partitions them according to whether they lead to successful or unsuccessful outcomes, and constructs a contrastive teacher distribution over future tokens. The strong model is then trained to match the distribution associated with successful repairs. This contrastive distillation yields consistent gains over direct imitation baselines across mathematical reasoning benchmarks, including larger improvements on harder subsets such as AIME 2025 and Omni-Hard.
Why it matters
This paper is highly relevant for AI researchers in the Netherlands focusing on LLM reasoning, alignment, and compute-efficient training. The proposed weak-to-strong distillation method offers actionable insights for Dutch AI labs aiming to enhance model performance without relying solely on massive scaling.
