Function-Level Execution Feedback for Code Preference Optimization
06:00 · August 26, 2026 · arXiv cs.AI RSS
Process supervision has improved mathematical reasoning, where intermediate steps are naturally expressed as chains of thought. In code generation, however, process supervision remains underexplored because there is no standard notion of a step. Supervision can target lines, reasoning traces, or program states, making it unclear what to label and optimize. We propose STEP-KTODER, a framework for code preference optimization that defines steps as module-level functions in decomposed multi-function programs and assigns binary correctness labels via automatically generated unit tests. Our method provides a code-specific instantiation of stepwise KTO, combining function-level process supervision with outcome-level feedback on the full program. We evaluate on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, showing that STEP-KTODER improves over outcome-only KTO and DPO. Further analysis shows that execution-based labels are essential: LLM-as-a-judge annotations systematically over-predict function failures, corrupt positive step labels, and degrade downstream preference optimization. Code is available at: https://github.com/inechnech/STEP-KTODER.
Summary
The paper addresses a key limitation in aligning large language models for code generation. While process supervision has strengthened mathematical reasoning by labeling intermediate steps, code lacks a comparable standard for what constitutes a step. Existing preference optimization methods such as DPO and KTO therefore operate only at the level of complete programs, supplying a coarse signal that marks an entire output as undesirable even when most of its components are correct.
STEP-KTODER resolves this ambiguity by treating module-level functions as the atomic steps. Reference solutions are first decomposed into multi-function programs. Automatically generated unit tests then assign binary correctness labels to each function, while the original dataset tests continue to supply an outcome label for the full program. These labels are combined in a joint objective that extends KTO with an additional term for function-level supervision, reinforcing locally correct functions and penalizing locally incorrect ones during offline preference optimization.
Experiments on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench show consistent gains over outcome-only KTO and DPO baselines, including improvements of more than 25 percentage points on the hardest subsets of BigCodeBench and LiveCodeBench for already instruction-tuned models. Additional analysis indicates that execution-based labels are essential: replacing them with LLM-as-a-judge annotations systematically over-predicts function failures, corrupts positive step labels, and ultimately weakens the resulting policy.
Why it matters
This research provides a highly actionable and novel methodology for aligning code generation models, which is directly applicable to Dutch AI researchers and software-heavy enterprises. The open-source nature and rigorous mathematical foundation make it an excellent resource for advanced AI practitioners in the Netherlands looking to improve LLM coding capabilities.
