LeRobot v0.6.0: Imagine, Evaluate, Improve

Summary

LeRobot v0.6.0 expands the open-source robotics toolkit with three world-model policies that incorporate future prediction during training. VLA-JEPA trains a compact Qwen3-VL-2B vision-language-action model to anticipate latent frames from its own actions, discarding the world-model component at inference so supervision adds no runtime overhead. LingBot-VA extends this idea into an autoregressive video-action model that generates future frames and actions in chunks while feeding real observations back for grounding, with optional video saving for inspection on a single 24–32 GB GPU. FastWAM pairs a large video-generation expert with a compact action expert inside one network, allowing the model to learn from its own imagined rollouts yet dropping the dreaming step entirely at deployment.

The release also integrates five additional VLAs. GR00T N1.7 updates the NVIDIA cross-embodiment model with a Cosmos-Reason2-2B backbone and flow-matching action head, maintaining parity with the original Isaac-GR00T implementation while making flash-attention optional. MolmoAct2, EO-1, EVO1, and the Multitask Diffusion Transformer each arrive with documented fine-tuning paths, checkpoint availability, and hardware footprints ranging from roughly 12 GB for inference to single 24 GB GPUs for LoRA adaptation.

A new unified reward-models interface adds Robometer, a 4B-parameter model pretrained on over a million trajectories to score progress and success from video and language, and TOPReward, a zero-shot wrapper that uses token log-probabilities from any capable VLM. Both integrate with labeling scripts that embed per-frame annotations directly into datasets for reward-aware training or quality checks.

Dataset handling gains depth-map support for RealSense cameras, stored as compact 12-bit streams and decoded to metric units at training time, plus an automatic VLM annotation pipeline that populates timestamped language labels, subtasks, and per-camera VQA pairs. Video encoding is now fully configurable, and data loading achieves up to 2× speedups through parallel multi-camera decoding, reduced inter-process memory, and persistent worker caches.

Evaluation consolidates six new simulation environments under a single lerobot-eval CLI alongside existing suites, each accompanied by Docker images and baseline checkpoints. Deployment shifts to a dedicated lerobot-rollout command that supports DAgger-style human corrections, continuous recording, and episode highlighting. Training adds FSDP sharding across GPUs with resumable single-file checkpoints and direct submission to Hugging Face Jobs for cloud execution on pay-as-you-go instances.

Why it matters

Directly actionable tooling and research updates for ML engineers working on robotics policies, benchmarks, and deployment pipelines; addresses production constraints like GPU memory, latency via Real-Time Chunking, and human-in-the-loop data collection.