Latent Goal Prediction from Language for Model-Based Planning
06:00 · June 23, 2026 · arXiv cs.AI RSS
Planning with world models is bottlenecked by compounding prediction errors and the difficulty of defining optimizable goals. Visual targets provide precise local gradients but poor distant guidance, while language is flexible yet limited by noisy cross-modal alignment or dependence on large generative models unsuited for the high-sampling nature of model-based planning. To address these challenges, we introduce Latent Goal Prediction from Language (LAGO), a framework that predicts both sequences of intermediate goal states from language instructions and action-conditioned rollouts, all within the same latent space. Rather than optimizing toward a single global objective, LAGO dynamically decomposes instructions into explicitly predicted, locally tractable latent subgoals. By updating these subgoals online and using a soft minimum trajectory cost during planning, LAGO enables an agent to follow coherent latent trajectories over long horizons. Evaluation across multiple environments planning horizons shows that LAGO avoids the sharp degradation of prior methods. By achieving robust and precise long-horizon planning purely from language, LAGO bridges the precision of visual goals with the flexibility of text-guided control.
Summary
Planning with world models is constrained by two persistent difficulties: prediction errors that compound over extended rollouts and the challenge of specifying goals that remain both optimizable and semantically meaningful. Visual targets supply precise local gradients yet offer little directional signal when the objective lies beyond the immediate planning horizon. Language instructions, by contrast, provide flexible task specification but typically introduce noisy cross-modal objectives or require large generative models that are ill-suited to the repeated sampling demanded by model-based planners.
LAGO addresses these limitations by learning to map natural language instructions directly into sequences of intermediate latent states inside the world model’s representation space. Rather than optimizing toward a single distant target, the framework predicts a series of locally reachable subgoals conditioned on the current language command. These subgoals are regenerated online at each planning step, and a soft-minimum trajectory cost encourages imagined rollouts to align with the evolving sequence. Because all operations occur in the same latent space used for dynamics prediction, the method avoids both the non-convexity of contrastive vision-language scores and the computational overhead of external generative models.
Evaluations across multiple environments and varying planning horizons show that LAGO maintains performance where earlier approaches degrade sharply once task length exceeds the reliable prediction horizon. By converting high-level textual directives into structured, continually updated latent trajectories, the approach combines the local precision of image-based goals with the compositional flexibility of language, enabling coherent long-horizon planning without explicit subgoal completion detection or privileged geometric information.
Why it matters
This research is highly relevant for AI researchers and practitioners in the Netherlands, particularly those focused on robotics, autonomous systems, and logistics. The LAGO framework offers actionable methodologies for improving long-horizon planning and text-guided control, aligning well with the Dutch high-tech sector's focus on advanced automation.
