iFLYTEK-Embodied-Omni Technical Report
06:00 · July 7, 2026 · arXiv cs.AI RSS
General-purpose embodied agents must understand multimodal instructions, anticipate how their environment will evolve, and produce precise control actions over extended horizons. Existing approaches typically specialize in visual-language reasoning, video-based world modeling, or action generation, while cascaded pipelines that first synthesize future observations and then infer actions can introduce interface bottlenecks and compound prediction errors. We present iFLYTEK-Embodied-Omni, a unified multimodal foundation model that jointly models vision(videos and images), language, and action within a single Omni framework. Its modality-specific visual-language, video-generation, and action-generation components communicate through shared multimodal self-attention. This design establishes brain-cerebellum collaboration: the vision-language modeland video generation model form a high-level brain for instruction understanding, task planning, progress tracking, and future visual-state prediction, whereas the action generation modelserves as a low-level cerebellum that directly converts planned subgoals and shared multimodal context into executable action chunks. To develop these capabilities, we combine action-annotated and action-free embodied videos from human demonstrations and robot interactions with embodied reasoning, embodied perception, and general-purpose image-text data to construct a comprehensive dataset. We further adopt a four-stage strategy that progressively trains the VLM, VGM, and AGM before jointly fine-tuning the complete model.
Summary
iFLYTEK-Embodied-Omni is presented as a single multimodal foundation model that jointly processes vision in the form of images and video, language instructions, and action sequences for embodied agents. Unlike prior systems that isolate visual-language reasoning, video-based world modeling, or action generation, or that chain separate modules in a pipeline, the model uses shared multimodal self-attention to let its components exchange information directly. This architecture is described as brain-cerebellum collaboration: a vision-language model together with a video-generation model acts as the high-level brain responsible for interpreting instructions, planning tasks, tracking progress, and predicting future visual states, while a dedicated action-generation model functions as the low-level cerebellum that translates planned subgoals and the shared context into executable action chunks.
The training regimen follows a four-stage schedule. The vision-language, video-generation, and action-generation components are first developed individually, then the full model is jointly fine-tuned. Data for this process combine action-annotated and action-free embodied videos drawn from both human demonstrations and robot interactions, supplemented by embodied reasoning and perception examples as well as general-purpose image-text pairs. The resulting dataset supports the model’s ability to handle multimodal instructions, anticipate environmental changes, and generate precise control signals over extended time horizons without the interface bottlenecks or accumulating prediction errors typical of cascaded approaches.
Why it matters
This technical report is highly relevant for AI researchers and robotics engineers in the Netherlands, offering a novel, unified architecture for embodied AI that overcomes the limitations of cascaded pipelines. It provides advanced methodologies for integrating multimodal inputs into actionable robotic controls, directly applicable to Dutch R&D in autonomous systems.
