Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR
06:00 · June 25, 2026 · arXiv cs.AI RSS
Reinforcement learning with verifiable rewards (RLVR) has been extended from single-domain training to multi-domain reasoning suites spanning mathematics, programming, and science. However, the training curriculum (how often each domain is sampled) is typically fixed or hand-tuned, even though reasoning skills transfer unevenly across domains. Existing learnability-based curricula adapt to where the policy is currently improving, but are blind to whether a gradient step on the selected domain benefits the remaining domains. In this paper, we propose Transfer-Aware Curriculum (TAC), a bandit-style online curriculum that prioritizes domains whose updates broadly benefit the rest of the training suite. TAC repurposes signals already produced by RL training: per-domain advantages capture local learnability, and projected gradients, taken from the GRPO step being computed, estimate cross-domain transferability via gradient-geometry alignment, at negligible cost (<1% wall-clock overhead). Across a six-domain reasoning suite, TAC achieves the best macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B, outperforming proportional random sampling, a hand-designed schedule, and a learnability-only bandit, and improving over the last of these by up to 2.8 points (10% relative). Ablations show performance degrades sharply when the transferability term is removed, and TAC remains robust on imbalanced training mixtures where learnability-only curricula over-commit to dominant domains. Our findings establish cross-domain transferability as a key signal for curriculum design in multi-domain RLVR.
Summary
Reinforcement learning with verifiable rewards has moved beyond single-domain setups to multi-domain suites that combine mathematics, programming, logic, simulation, tables, and science. In these settings the sampling distribution over domains is still commonly fixed or hand-tuned, even though empirical transfer matrices show that reasoning gains from one domain reach the others unevenly. Existing learnability-driven curricula react only to local advantage signals and therefore risk over-sampling domains whose updates remain narrowly scoped.
Transfer-Aware Curriculum addresses this gap by casting domain selection as a multi-armed bandit whose per-arm score fuses two quantities already computed during Group Relative Policy Optimization. Per-domain advantages supply the conventional learnability term; projected gradients maintained as exponential moving averages supply an additional transferability term that measures alignment, via cosine similarity, between a candidate domain’s update direction and the directions observed on the remaining domains. The two signals are combined with a single mixing coefficient and incur less than one percent extra wall-clock cost.
On a six-domain benchmark suite the resulting schedule raises macro-averaged accuracy on both Qwen3-1.7B and Llama3.2-3B relative to proportional random sampling, a hand-designed math-heavy schedule, and a learnability-only bandit, with gains reaching 2.8 points. Ablations confirm that removing the transferability component sharply degrades performance, while the full method remains stable under skewed data budgets that cause purely learnability-based curricula to over-commit to dominant domains.
Why it matters
This research provides Dutch AI researchers and developers with an efficient, novel methodology for training multi-domain reasoning models. Improving cross-domain transferability in RLVR can help Dutch AI enterprises and academic labs optimize model training and computational resource allocation.
