Interactive World Simulation with Cosmos for Physical AI

Cosmos3 Examples Collection | Code | Technical Report | Website

Model Overview: Cosmos3-Nano-Sim-Transfer

The highlighted checkpoint, Cosmos3-Nano-Sim-Transfer, is a post-trained, causal-distilled few-step variant of the Cosmos3-Nano base model.

Description

Depth and segmentation: Controls on top, corresponding generated RGB videos below.

Edge and blur: Controls on top, corresponding generated RGB videos below.

Cosmos3-Nano-Sim-Transfer generates photorealistic RGB video conditioned on simulator-rendered control sequences, such as depth maps, segmentation masks, edge maps, or blurred images, and a text prompt. It is designed for real-time integration with simulators such as NVIDIA Isaac Lab, generating visual observations that preserve the scene geometry and dynamics specified by the simulator controls. Policies developed in these simulators can be trained and evaluated using the generated photorealistic observations while the underlying simulation and its ground-truth state remain unchanged. The simulator remains responsible for physics, scene state, and ground-truth annotations, while the model generates visual appearance conditioned on the simulator controls. Inference throughput depends on the hardware and generation configuration.

Cosmos3-Nano-Sim-Transfer was developed by NVIDIA as part of the Cosmos example model family.

For a quick start, see the Cosmos3-Nano-Sim-Transfer skill.

This model is ready for commercial or non-commercial use.

License

This model is released under OpenMDW1.1.

Deployment Geography

Global

Use Case

Physical AI for robotics: real-time sim-to-real visual transfer through photorealistic rendering of simulated environments for reinforcement learning (RL) training and policy evaluation in classical simulators such as NVIDIA Isaac Lab.

Release Date

• Hugging Face 10/20/2026 via https://huggingface.co/collections/nvidia/cosmos3-examples
• GitHub 10/20/2026 via https://github.com/nvidia/cosmos

Model Architecture

Architecture Type: Transformer

Network Architecture: Mixture-of-Transformers (MoT)

Cosmos3-Nano-Sim-Transfer is derived from Cosmos3-Nano through post-training for controlled video generation and causal distillation. The model generates video in small latent-frame chunks conditioned on cached history, incoming simulator-rendered controls, and a text prompt. Supported controls include depth maps, segmentation masks, edge maps, and blurred images. This generation process supports closed-loop policy training and evaluation while the simulator maintains the physical state and dynamics.

This model was developed using the Cosmos Framework.

Number of trainable model parameters: 16B

Inputs

• Input Type(s): Video (control video), Text
• Input Format(s):
  • Control video: Simulator-rendered depth maps, segmentation masks, edge maps, or blurred images for each frame
  • Text: String
• Input Parameters:
  • Control video: Three-dimensional (3D: time, height, width)
  • Text: One-dimensional (1D)
• Other Properties Related to Input:
  • Designed for integration into a simulator rendering loop (for example, NVIDIA Isaac Lab), with geometry and physics maintained by the simulator and visual appearance generated by the model.
  • Designed for real-time batched inference across up to eight parallel environments on a single GB200 GPU. Inference throughput depends on the batch size and generation configuration.

Outputs

• Output Type(s): Video
• Output Format(s):
  • Video: MP4 or streamed video frames
• Output Parameters:
  • Video: Three-dimensional (3D)
• Other Properties Related to Output:
  • Generated frames are conditioned on the input scene geometry. Simulator annotations describe the underlying simulated scene; their alignment with generated imagery should be validated for the intended use case.
  • Generates up to 600 frames at 480p with an output video frame rate of 30 frames per second (fps).
  • Output video frame rates describe playback timing; inference throughput is reported separately.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

Runtime Engine(s):

• PyTorch
• vLLM-Omni
• SGLang

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell
• NVIDIA Hopper

Operating System(s):

• Linux (other operating systems have not been tested).

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

Cosmos3-Nano-Sim-Transfer — V1.0 (10/20/2026)

Cosmos3-Nano-Sim-Transfer is a post-trained, causal-distilled variant of Cosmos3-Nano that supports few-step generation.

It can be integrated into a classical simulation pipeline by taking simulator-rendered control sequences as input and generating photorealistic video observations for reinforcement learning and policy evaluation, while the simulator remains responsible for physics and ground-truth state.

Training and Evaluation Datasets

Post-Training Datasets

Data Modality: Video and Text.

Post-Training Data Size:

• Text: 1 Billion to 10 Trillion Tokens
• Video: Less than 10,000 Hours

Data Collection Method: Hybrid: Automatic/Sensors, Synthetic, Automated.

Labeling Method: Hybrid: Manually-Labeled, Automated.

Properties: The post-training dataset consists of video clips with English captions and associated visual control annotations. The data includes real and synthetic content from an NVIDIA-curated dataset.

Dataset Overview

• Total Size: 1.3B data points
• Total Number of Datasets: 393 dataset entries
• Dataset partition: Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately]
• Time period for training data collection: 2024–2026
• Time period for validation data collection: N/A (standard public benchmarks)

Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins.

Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release.

For more information about the datasets used to train this model, please see the Public Summary of Training Content.

Data Modality and Training Data Size

Data Collection Method by dataset

• Hybrid: Automatic/Sensors, Synthetic, Automated

Labeling Method by dataset

• Hybrid: Manually-Labeled, Automated

Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.

Public Datasets

Private Datasets

Synthetic Datasets

Evaluation Datasets

Data Collection Method by dataset

• Hybrid: Automatic/Sensors, Synthetic, Automated

Labeling Method by dataset

• Hybrid: Manually-Labeled, Automated

Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.

Quality Benchmark(s)

PAI-Bench-C (TransferBench)

We compare Cosmos3-Nano-Sim-Transfer with Cosmos3-Nano on PAI-Bench-C (TransferBench) using DOVER to estimate video quality and four control-following metrics to evaluate control fidelity. We map generated videos into each control modality and compare the resulting representations with the supplied controls. Segmentation mean intersection over union (mIoU) measures mask overlap; blur structural similarity (SSIM) measures agreement after blurring; edge F1 combines precision and recall of edge agreement; and depth scale-invariant root mean squared error (SI-RMSE) measures depth differences after scale normalization.

Prompt following and temporal consistency

We report VLM-judge scores for prompt following and video quality. Perceptual drift evaluates changes in video quality over time (PAI-Bench-G). Temporal Sampson error evaluates temporal consistency by measuring geometric agreement between matched image points across frames.

Inference

Acceleration Engine: PyTorch, vLLM-Omni

Test Hardware: NVIDIA Blackwell GB200, NVIDIA Hopper H100, H200, H20, RTX PRO 6000

Inference Performance

Numbers are based on vLLM-Omni, using a single GPU. Empty table cells indicate that measurements are not reported.

Usage Example

Quick Start

This example serves the model with vLLM-Omni and generates a 600-frame depth-controlled video at 30 FPS and 640 × 640. The control video and prompt are published in assets/sample_data/example1.

1. Start the server

On a Linux host with an NVIDIA GPU (tested on one GB200), Docker, and the NVIDIA Container Toolkit:

docker run --runtime nvidia --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$PWD:$PWD:ro" \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env VLLM_OMNI_VIDEO_SYNC_TIMEOUT=7200 \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano-Sim-Transfer \
    --omni --model-class-name Cosmos3NanoSimTransferPipeline \
    --deploy-config /app/vllm-omni/vllm_omni/deploy/cosmos3_nano_sim_transfer.yaml \
    --no-guardrails \
    --host 0.0.0.0 --port 8000 --init-timeout 1800

The container publishes the server on 127.0.0.1:8000 of the host. --deploy-config selects the Sim-Transfer pipeline. --no-guardrails disables the Cosmos3 text and video safety checks. The first start downloads the checkpoint.

Default serving is eager. For faster repeated runs, enable regional compilation and use_cuda_graphs, and set vae_fast_path and vae_encode_fast_path to channels_last. The first request will be slower while these warm up.

For optimized server settings and more examples with depth, edge, blur, and segmentation controls, see the Sim-Transfer vLLM-Omni notebook.

2. Generate the depth video

Generate a 600-frame depth-controlled video using the supplied prompt and seed. The example uses a 30-frame history window with three attention sinks.

pip install requests huggingface_hub

import json
from pathlib import Path
from uuid import uuid4

import requests
from huggingface_hub import snapshot_download

MODEL = "nvidia/Cosmos3-Nano-Sim-Transfer"
example_dir = Path(snapshot_download(
    MODEL,
    allow_patterns=[
        "assets/sample_data/example1/example1_depth.mp4",
        "assets/sample_data/example1/example1_upsampled_prompt.json",
    ],
    local_dir=Path("Cosmos3-Nano-Sim-Transfer").resolve(),
)) / "assets/sample_data/example1"
metadata = json.loads((example_dir / "example1_upsampled_prompt.json").read_text())

extra_params = {
    "session_id": uuid4().hex,
    "reset": True,
    "close_session": True,
    "resolution": "480",
    "max_prompt_tokens": 4096,
    "kv_cache_inference_size": 30,
    "attention_sink_size": 3,
    "emphasize_control_in_prompt": False,
    "depth": {"control_path": str(example_dir / "example1_depth.mp4")},
}
response = requests.post(
    "http://127.0.0.1:8000/v1/videos/sync",
    data={
        "model": MODEL,
        "prompt": metadata["prompt"],
        "size": "640x640",
        "num_frames": "600",
        "fps": "30",
        "seed": str(metadata["seed"]),
        "extra_params": json.dumps(extra_params),
    },
    headers={"Accept": "video/mp4"},
    timeout=7260,
)
response.raise_for_status()
Path("robot_depth.mp4").write_bytes(response.content)

With the supplied depth control, the generated output looks like this:

Quick Start with SGLang

SGLang serves the same model through its OpenAI-compatible Videos API and generates the same depth-controlled video. Two things differ: the server reads the control video from a path it can open instead of accepting uploads, so the working directory is mounted into the container, and generation runs as an asynchronous job that the client polls. For edge, blur, and segmentation controls with SGLang, see the SGLang notebook.

1. Start the server

On the same kind of host, with the published lmsysorg/sglang:dev image, which includes the Cosmos3 pipelines from sgl-project/sglang#39475:

export SGLANG_HF_CACHE="${SGLANG_HF_CACHE:-$HOME/.cache/sglang-huggingface}"  # The container runs as root.
mkdir -p "$SGLANG_HF_CACHE"
chmod 777 "$SGLANG_HF_CACHE"
docker pull lmsysorg/sglang:dev
docker run --init --runtime nvidia --gpus all --ipc=host \
  -v "$SGLANG_HF_CACHE:/root/.cache/huggingface" \
  -v "$PWD:$PWD:ro" \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 127.0.0.1:30000:30000 \
  lmsysorg/sglang:dev \
  sglang serve \
    --model-path nvidia/Cosmos3-Nano-Sim-Transfer \
    --host 0.0.0.0 --port 30000

The container publishes the server on 127.0.0.1:30000 of the host and mounts the working directory, read-only, at the same path; the client below downloads the control video under that directory and names it by path. The dev tag moves with SGLang's main branch, so pull it before each start. The first start downloads the checkpoint; the HTTP port opens after the weights are loaded and the warmup request finishes. The distilled checkpoint fixes the step count and guidance, and the server reads its history window from the export's training configuration, so the request sets neither.

2. Generate the depth video

Run this from the directory mounted above:

pip install requests huggingface_hub

import json
import time
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

MODEL = "nvidia/Cosmos3-Nano-Sim-Transfer"
SERVER = "http://127.0.0.1:30000"
example_dir = Path(snapshot_download(
    MODEL,
    allow_patterns=[
        "assets/sample_data/example1/example1_depth.mp4",
        "assets/sample_data/example1/example1_upsampled_prompt.json",
    ],
    local_dir=Path("Cosmos3-Nano-Sim-Transfer").resolve(),  # Inside the mounted working directory.
)) / "assets/sample_data/example1"
metadata = json.loads((example_dir / "example1_upsampled_prompt.json").read_text())

form = {
    "model": MODEL,
    "prompt": metadata["prompt"],
    "size": "640x640",
    "num_frames": "600",  # Caps the control frames consumed; the server trims the clip to whole chunks.
    "fps": "30",
    "seed": str(metadata["seed"]),
    "control_hint": "depth",
    "control_path": str(example_dir / "example1_depth.mp4"),  # Opened by the server; nothing is uploaded.
    "extra_params": json.dumps({"emphasize_control_in_prompt": False}),
}
# Submit the job, poll it, and download the video. The form carries text fields only.
response = requests.post(f"{SERVER}/v1/videos", files=[(key, (None, value)) for key, value in form.items()], timeout=60)
response.raise_for_status()
job = response.json()
deadline = time.monotonic() + 7200
while job["status"] not in ("completed", "failed", "cancelled"):
    if time.monotonic() > deadline:
        raise TimeoutError(f"Job {job['id']} did not finish in time: {job}")
    time.sleep(10)
    job = requests.get(f"{SERVER}/v1/videos/{job['id']}", timeout=60).json()
if job["status"] != "completed":
    raise RuntimeError(json.dumps(job, indent=2))
video = requests.get(f"{SERVER}/v1/videos/{job['id']}/content", timeout=600)
video.raise_for_status()
Path("robot_depth.mp4").write_bytes(video.content)

The output follows the control clip, trimmed to the pipeline's whole latent chunks, so it can be a few frames shorter than the requested count.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.