Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models
06:00 · July 7, 2026 · arXiv cs.AI RSS
Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge. Conventional refusal-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries. Building upon the constructive safety paradigm pioneered by Oyster-I, which moves beyond blanket refusal toward thoughtful, response-oriented safety alignment, we identify two critical limitations of its Supervised Fine-Tuning (SFT)-based scheme: insufficient safety generalization to out-of-distribution scenarios and a phenomenon we term safety chain-of-thought (CoT) over-generalization, wherein safety-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience. To address these limitations, we propose Oyster-II, a reinforcement learning (RL)-based constructive safety alignment framework that adopts a Zero-RL paradigm combined with a multi-stage reinforcement learning strategy.Evaluated across extensive benchmarks, Oyster-II comprehensively surpasses both Qwen3-14B and its predecessor Oyster-I on safety dimensions, achieving cross-scale performance comparable to Qwen3-Max and Qwen3.5-397B.
Summary
Large language models continue to face the dual requirement of blocking genuinely harmful outputs while still addressing legitimate user intent in sensitive queries. Conventional refusal-based alignment often withholds information that could be provided safely, prompting a shift toward constructive safety approaches that deliver partial, risk-aware responses. Oyster-I introduced this response-oriented paradigm, yet its supervised fine-tuning foundation proved limited in generalizing to out-of-distribution safety scenarios and introduced safety chain-of-thought over-generalization, in which safety reasoning patterns were applied indiscriminately to benign inputs and reduced overall helpfulness.
Oyster-II replaces the SFT stage with a Zero-RL paradigm and a multi-stage reinforcement learning pipeline. The framework incorporates length-reward entropy control and benign-sample length regulation to avoid reward hacking and premature convergence. It further introduces SERL, an algorithm extending GSPO with a mix-policy strategy that accelerates convergence and strengthens adherence to instruction hierarchies between developer policies and user requests. A curriculum-based training schedule combined with active-learning difficulty control mitigates reward noise, while long-context safety alignment on extended queries improves semantic understanding and reduces over-refusal driven by shallow keyword matching.
Evaluations across multiple benchmarks show Oyster-II outperforming both its predecessor and the Qwen3-14B base model on safety metrics while matching the performance of substantially larger systems such as Qwen3-Max and Qwen3.5-397B. These gains occur without invasive changes to the base model, preserving general capabilities and linguistic style. The approach therefore offers a scalable route to models that maintain clear safety boundaries alongside practical utility.
Why it matters
This research is highly relevant to the Dutch AI market due to the Netherlands' and EU's strong regulatory focus on ethical, safe, and transparent AI. Oyster-II provides advanced researchers with actionable RL methodologies to align LLMs safely without compromising their utility, directly supporting compliant AI development.
