Interpreting Latent CoT Reasoning as Dynamical Systems
06:00 · July 14, 2026 · arXiv cs.AI RSS
Recent latent reasoning methods, such as CODI and COCONUT, face a fundamental interpretability problem: they maintain multiple superimposed candidate traces in the hidden space at each step, unlike explicit- CoT, which follows a single transparent reasoning trace. Existing mechanistic methods show compression, shortcuts, and superposition without explaining how reasoning evolves across latent steps. To address this gap, we model latent token sequences as trajectories in representation space and apply dynamical systems analysis to characterize the evolution of reasoning. Using quantitative measures, such as step-to-step change, direction consistency, and Lyapunov sensitivity, alongside qualitative projections, such as UMAP and DMD/PHATE, we show that latent CoT exhibits structured, non-random dynamics with two distinct stability classes. CODI behaves as a stable attractor, while COCONUT behaves as an unstable expanding system, and SIM-CoT supervision tightens both behaviors without changing the underlying dynamics. This framework advances the interpretability of latent CoT reasoning dynamics and provides actionable insights for improving latent reasoning performance. Code1 and Project page2 available online.
Summary
This paper presents a dynamical-systems framework for interpreting latent Chain-of-Thought reasoning in models such as CODI and COCONUT. Rather than treating successive hidden states as isolated vectors, the authors model the sequence of latent representations as a trajectory in representation space. The approach draws on quantitative measures of step-to-step displacement, directional consistency, and Lyapunov sensitivity, together with qualitative projections obtained through UMAP, DMD, and PHATE, to characterize how reasoning states evolve across latent steps.
Applied to GSM8K trajectories, the analysis distinguishes two stability regimes. CODI produces trajectories that converge toward a stable attractor, with limited divergence between successive states. COCONUT, by contrast, generates expanding trajectories whose eigenvalues indicate instability and growing separation. When both architectures are trained under the SIM-CoT regime, which adds an auxiliary decoder to align latent states with explicit reasoning steps, the magnitude of change and directional variance decrease, yet the underlying attractor or expansion pattern remains unchanged.
The framework therefore supplies a concrete language for describing the latent instability problem observed in prior work: compression, shortcut pathways, and representational collapse appear as geometric consequences of these distinct dynamical regimes. By grounding interpretability in measurable trajectory properties rather than post-hoc inspection of individual tokens, the method offers a route to diagnose and potentially regularize latent reasoning without altering its core computational structure.
Why it matters
The article is highly relevant for AI researchers in the Netherlands focusing on LLM interpretability and trustworthy AI. Understanding the internal dynamics of latent reasoning aligns strongly with EU and Dutch priorities for transparent and explainable AI systems.
