nvidia-cosmos / cosmos-cookbook

Cosmos Cookbook A comprehensive guide for working with the **NVIDIA Cosmos ecosystem**—a suite of World Foundation Models (WFMs) for real-world, domain-specific applications across robotics, simulation, autonomous systems, and physical scene understanding. **📚 View the Full Documentation →** — Step-by-step workflows, case studies, and technical recipes Latest Updates | **Date** | **Recipe** | **Model** | **Description** | |----------|------------|-----------|-----------------| | Mar 16 | Cosmos-Reason2 on Jetson Thor for Edge VLM Perception | Cosmos Reason 2 | Deploy Cosmos-Reason2 on Jetson AGX Thor for social robots (IntBot), with FP8 quantization and TensorRT-Edge-LLM optimization | | Mar 15 | Content-Adaptive Video Compression for Cosmos Curator with Beamr CABR | Cosmos Curator | Recipe for replacing Cosmos Curator's default CPU-based, fixed-bitrate video encoder with Beamr CABR (Content-Adaptive Bitrate) — a GPU-accelerated video optimization and encoding solution | | Mar 15 | Post-Training Cosmos-H-Surgical-Simulator for Surgical Robotics | Cosmos Predict 2.5 | Fine-tune Cosmos Predict 2.5 as an action-conditioned surgical simulator for policy evaluation and synthetic data generation using the SutureBot dataset | | Mar 15 | Outlier Detection in Embedding Vector Trajectories | Cosmos Curator | Outlier detection in video embedding trajectories via Time Series K-Means + Soft-DTW distance thresholding | | Mar 3 | GR00T-Dreams: Synthetic Trajectory Generation for Robot Learning | Cosmos Predict 2.5, Reason 2 | End-to-end pipeline for synthetic robot trajectory generation: post-train Predict 2.5 on GR1 data, generate trajectories, and use Cosmos Reason 2 as video critic for rejection sampling | | Feb 18 | Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning | Cosmos Predict 2.5 | Recipe upgraded to **Cosmos Predict 2.5**: state-of-the-art robot policy via latent frame injection. Results—LIBERO 98.33%, RoboCasa **71.1%** (new SOTA, +4% over Predict2) | | Feb 18 | 3D AV Grounding Post-Training with Cosmos Reason 1 & 2 | Cosmos Reason 1 & 2 | 3D vehicle grounding in autonomous driving: detect and localize vehicles in 3D from camera images with SFT (Cosmos-RL and Qwen-Finetune) | | Feb 4 | Worker Safety in a Classical Warehouse | Cosmos Reason 2 | Zero-shot industrial safety compliance and hazard detection in classical warehouse environments using context-aware prompt engineering | | Jan 30 | Prompt Guide | Cosmos Reason 2 | Inference Prompt Guide | Upcoming Activities NVIDIA GTC 2026 Register for NVIDIA GTC happening **March 16–19, 2026**, and add the Cosmos sessions to your calendar. Don't miss the must-see keynote from CEO Jensen Huang at SAP Center on Monday, March 16 at 11:00 a.m. PT. NVIDIA Cosmos Cookoff Introducing the **NVIDIA Cosmos Cookoff** — a virtual, four-week physical AI challenge running **January 29 – February 26** for robotics, AV, and vision AI builders. Build with NVIDIA Cosmos Reason and Cosmos Cookbook recipes—from egocentric robot reasoning to physical plausibility checks and traffic-aware models for a chance to win **$5,000**, an **NVIDIA DGX Spark**, and more! **Register Now →** Sponsored by Nebius and Milestone. Prerequisites | Use Case | Linux (Ubuntu) | macOS | Windows | |----------|----------------|-------|---------| | Running cookbook recipes (GPU workflows) | ✅ Supported | ❌ | ❌ | | Local documentation & contribution | ✅ Supported | ✅ Supported | ⚠️ WSL recommended | For Documentation & Contribution (All Platforms) • **Git** with Git LFS • **Python**: Version 3.10+ • **Internet access** for cloning and dependencies For Running Cookbook Recipes (Ubuntu Only) Full GPU workflows require an Ubuntu Linux environment with NVIDIA GPUs. → See **Getting Started** for complete hardware and software requirements. → Or **Deploy on Cloud** (Nebius, Brev, and more to come) for ready-to-launch GPU instances. Quick Start • Install Git LFS (Required) > ⚠️ **Important**: This repository contains many media files (videos, images, demonstrations). Git LFS is **required** to clone and work with this repository properly. For other platforms (macOS, Windows, Fedora), see the official installation guide at **git-lfs.com**. If you've already cloned without LFS, fetch the media files with: • Install System Dependencies For other platforms, see **astral.sh/uv** for installation instructions. • Clone and Setup Repository • Explore the Documentation Then open http://localhost:8000 in your browser. Repository Structure The Cosmos Cookbook is organized into two main directories: Contains the source documentation in markdown files: • Technical guides and workflows • End-to-end examples and case studies • Step-by-step recipes and tutorials • Getting started guides Contains executable scripts referenced throughout the cookbook: • Data processing and curation pipelines • Model evaluation and quality control scripts • Configuration files for post-training tasks • Automation tools and utilities This structure separates documentation from implementation, making it easy to navigate between reading about workflows and executing the corresponding scripts. Media File Guidelines When contributing media files, prefer over : • **Better quality** — MP4 supports full color depth vs GIF's 256-color limit • **Smaller file size** — Modern video codecs compress far more efficiently • **Audio support** — MP4 can include narration when needed Use **H.264** encoding for universal browser compatibility. Available Commands Contributing & Support • **Contributing Guide** - How to contribute to the cookbook • **Report Issues**: Use GitHub Issues for bugs and feature requests • **Share Success Stories**: We love hearing how you use Cosmos models creatively License and Contact This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use. NVIDIA Cosmos source cod…