Internalizing the Future: A Unified Agentic Training Paradigm for World Model Planning
06:00 · June 29, 2026 · arXiv cs.AI RSS
Large language model (LLM) agents have demonstrated strong capability in sequential decision-making, yet they remains fundamentally reactive in long-horizon tasks. Unlike humans who employ "what-if" reasoning to evaluate potential plans before commitment, standard agents lack an internal world model to simulate future outcomes. Therefore, we propose to internalize future-aware planning by training a single autoregressive model to verbalize both a prospective state rollout and a plan-conditioned success estimate-a textual analogue of the Q-value. Crucially, we identify a format-capability gap: simply fine-tuning agents on look-ahead traces during post-training leads to superficial mimicry of foresight without genuine predictive grounding. To bridge this gap, we introduce a three-stage training paradigm: (i) World Model Agentic Mid-Training (WM-AMT) to inject latent predictive capabilities into the policy; (ii) Format-Eliciting SFT (FE-SFT) to structure this injected capability; and (iii) Foresight-Conditioned Reinforcement Learning (FC-RL) to refine the calibration and utility of the generated simulations. Evaluated on search and mathematical reasoning tasks, our approach consistently outperforms other training baselines. Our results demonstrate that effective internal world modeling in LLM agents requires a capability-first training pipeline to achieve grounded and calibrated foresight.
Summary
Large language model agents already handle sequential decision-making tasks with reasonable competence, yet they remain largely reactive when tasks stretch over many steps. Standard training leaves them without an internal mechanism to simulate how the world might evolve under different plans, a capability humans routinely apply through “what-if” reasoning before acting. The work addresses this limitation by training a single autoregressive model to produce two outputs in one forward pass: a prospective rollout of future states and a plan-conditioned estimate of success, the latter serving as a textual counterpart to a Q-value.
A central observation is that simply fine-tuning on foresight traces during ordinary post-training produces only superficial imitation. The model learns to echo the expected format without acquiring genuine predictive grounding, a phenomenon the authors term the format-capability gap. To close this gap they introduce a three-stage pipeline that first builds latent predictive ability, then organises it into usable form, and finally aligns it with task performance.
The first stage, World Model Agentic Mid-Training, injects predictive representations into the policy before any format supervision is applied. The second stage, Format-Eliciting Supervised Fine-Tuning, elicits structured verbalisations of those representations. The third stage, Foresight-Conditioned Reinforcement Learning, further calibrates the generated simulations so that their estimated success values become reliable guides for action selection. When evaluated on search and mathematical reasoning benchmarks, agents trained under this regime outperform standard baselines, indicating that internal world modelling in LLM agents benefits from an explicit capability-first sequence rather than direct imitation of look-ahead behaviour.
Why it matters
This research is highly relevant for AI researchers and advanced practitioners in the Netherlands developing autonomous LLM agents. The proposed training paradigm offers actionable methodologies to overcome the reactive limitations of current agents, aligning with the Dutch focus on advanced, capable, and reliable AI systems.
