Tandem Reinforcement Learning with Verifiable Rewards
06:00 · June 29, 2026 · arXiv cs.AI RSS
Reinforcement learning with verifiable rewards (RLVR) has significantly improved the reasoning capability of large language models, reaching expert or even superhuman performance in domains such as competition math. However, whether weaker agents and humans can actually harness this capability is far less certain, with RLVR documented to drift reasoning toward idiosyncratic patterns such as poor readability and language mixing. Tandem training is a recently introduced paradigm that targets this compatibility problem: a trained, stronger senior co-generates each rollout with a frozen, weaker junior, and the two are rewarded as a team, so the senior is pushed to reason in ways the junior can follow. Yet this paradigm has so far been demonstrated only in proof-of-concept settings, leaving open whether it scales to the long chains of thought of the modern RLVR pipeline. In this work, we propose Tandem Reinforcement Learning (TRL), which carries the tandem training paradigm into RLVR. In TRL, the senior and a frozen junior alternate stochastically to co-generate the reasoning, the resulting generation is rewarded, and the standard GRPO loss is applied to the senior. Training Qwen3-4B-Instruct on competition math, we find that TRL matches vanilla GRPO on solo reasoning capability while three properties emerge together from the same rollout structure: stronger handoff robustness with the junior, reduced distributional drift from the junior, and a chain-of-thought more legible to the junior. Our results demonstrate a promising route for RLVR with practical payoffs in multi-model communication and human compatibility.
Summary
Reinforcement learning with verifiable rewards has driven substantial gains in large language model reasoning on tasks such as competition mathematics, yet it often produces chains of thought that drift from the base model’s distribution. This drift manifests as reduced readability, language mixing, and token patterns that weaker models or human readers struggle to follow or continue. Tandem training was introduced to address compatibility by having a stronger senior model co-generate trajectories with a frozen junior partner, so that the team reward encourages the senior to stay within the junior’s predictive reach.
Tandem Reinforcement Learning extends this approach to full-scale RLVR pipelines. In TRL, a trainable senior and a frozen junior, both starting from the same base checkpoint, alternate stochastically at word boundaries while producing each rollout. The completed trajectory receives the standard binary verifier reward, and the senior is updated with the usual Group Relative Policy Optimization loss applied only to the tokens it emitted. Because the reward function, verifier, and loss remain unchanged, performance differences relative to vanilla GRPO can be attributed directly to the altered rollout structure.
When Qwen3-4B-Instruct is trained on competition math under this regime, solo benchmark performance matches that of a matched GRPO baseline. At the same time, three compatibility properties appear together: handoff robustness improves, with the TRL senior outperforming the GRPO senior by up to 6.6 pass@8 points when paired with the frozen junior at inference; distributional drift is reduced, shown by a 14 percent lower KL divergence from the base model and by 87 percent of the tokens most displaced under GRPO shifting back toward the base; and the generated reasoning becomes more legible to the junior, reflected in up to 17 percent lower per-token cross-entropy. An ablation that adds an explicit per-token KL penalty toward the junior does not recover these gains, confirming that the tandem rollout structure itself is responsible.
The results indicate that a single change to how rollouts are assembled can simultaneously preserve reasoning capability and improve compatibility with weaker agents, without requiring an explicit specification of intelligible reasoning in advance.
Why it matters
Novel primary research on RL for LLMs with technical depth and clear implications for multi-agent compatibility and human-AI alignment, directly applicable by Dutch AI researchers working on ethical, transparent systems.
