Interactive World Simulation with Cosmos for Physical AI

Cosmos3 Examples Collection | Code | Technical Report | Website

Model Overview: Cosmos3-Nano-Transfer-Auto

The highlighted checkpoint, Cosmos3-Nano-Transfer-Auto, is a post-trained variant of the Cosmos3-Nano base model.

Description

Cosmos3-Nano-Transfer-Auto generates synchronized multi-view camera videos and LiDAR observations. It supports up to 11 standard and fisheye camera views. Generation is conditioned on a text prompt, World Scenario Map (WSM) controls for each camera view, and aligned LiDAR controls when LiDAR output is requested. The WSM inputs encode driving-scene structure, including traffic signs, traffic lights, lane markings, and object cuboids, while the text prompt specifies appearance attributes such as weather, lighting, and time of day. This enables the same driving scenario to be regenerated at scale with controlled variations in scene appearance. The model uses bidirectional generation for batch synthetic data generation and serves as the source checkpoint for causal distillation into Cosmos3-Nano-Sim-Auto, which provides closed-loop simulation with the same multi-view, multi-sensor output modalities.

Cosmos3-Nano-Transfer-Auto was developed by NVIDIA as part of the Cosmos example model family.

For a quick start, see the Cosmos3-Nano-Transfer-Auto skill.

This model is ready for commercial or non-commercial use.

License

This model is released under OpenMDW1.1.

Deployment Geography

Global

Use Case

Physical AI for autonomous vehicles: synchronized multi-sensor synthetic data generation at scale for training and evaluating perception and driving models.

Release Date

• Hugging Face 10/20/2026 via https://huggingface.co/collections/nvidia/cosmos3-examples
• GitHub 10/20/2026 via https://github.com/nvidia/cosmos

Model Architecture

Architecture Type: Transformer

Network Architecture: Mixture-of-Transformers (MoT)

Cosmos3-Nano-Transfer-Auto is derived from Cosmos3-Nano through post-training on multi-sensor driving data, including LiDAR. The model jointly denoises the full multi-view sequence using bidirectional temporal context, conditioned on the text prompt, WSM camera controls, and aligned LiDAR controls when generating LiDAR. It produces synchronized camera videos for up to 11 views and corresponding LiDAR observations.

This model was developed using the Cosmos Framework.

Number of trainable model parameters: 16B

Inputs

• Input Type(s): Text; Video (World Scenario Map (WSM) camera controls); Image (optional initial RGB conditioning); Other: numerical LiDAR controls and optional measured LiDAR conditioning.
• Input Format(s):
  • Text: String (scene description: weather, lighting, time of day; also carries the camera configuration)
  • World Scenario Map (WSM) camera controls: Synchronized MP4 videos of rendered scene structure, including lane markings and object cuboids, one per selected camera view.
  • LiDAR controls: Prepared HD-map range-map sequences aligned with the camera controls, supplied in the supported range-map archive or tensor format. Required when generating LiDAR.
  • Optional RGB conditioning: Synchronized initial camera images, supplied directly or extracted from the first frame of each camera video.
  • Optional LiDAR conditioning: An initial measured LiDAR sweep in the supported range/intensity representation, aligned with the start of the camera and control sequences.
• Input Parameters:
  • Text: One-dimensional (1D)
  • WSM camera controls: Three-dimensional (3D), per view
  • Optional RGB conditioning: Two-dimensional (2D), per image
  • HD-map/LiDAR controls: Three-dimensional (3D) sequences of projected range maps: time × height × width.
  • Optional LiDAR conditioning: Two-dimensional (2D) range-view map per initial sweep: height × width, with range, intensity, and validity channels.
• Other Properties Related to Input:
  • Supported camera rigs contain 1–11 views (Hyperion 8.1 camera set: including front-wide 120°, front-tele 30°, cross-left/right 120°, rear-left/right 70°, rear-tele 30°, and front/left/right/rear fisheye 200°) and 128-beam rotating LiDAR. Adaptation to other rigs requires additional post-training.
  • WSM controls are produced by a WSM renderer; see the Cosmos3-Nano-Transfer-Auto skill for details.

Outputs

• Output Type(s): Video (multi-view, synchronized), LiDAR
• Output Format(s):
  • Video: MP4, one stream per camera view (standard and fisheye)
  • LiDAR: Range and intensity maps, converted to 3D point clouds with per-point intensity
• Output Parameters:
  • Video: Three-dimensional (3D), per view
  • LiDAR: Three-dimensional (3D)
• Other Properties Related to Output:
  • Camera videos and LiDAR observations are spatially and temporally synchronized.
  • Generates up to 300 frames per view at 480p or 720p, with output video frame rates of 10–30 frames per second (fps).
  • Output video frame rates describe playback timing; inference throughput is reported separately.

Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g., GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.

Software Integration

Runtime Engine(s):

• PyTorch
• vLLM-Omni
• SGLang

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell
• NVIDIA Hopper

Operating System(s):

• Linux (other operating systems have not been tested).

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

Model Version(s)

Cosmos3-Nano-Transfer-Auto — V1.0 (10/20/2026)

Cosmos3-Nano-Transfer-Auto is a post-trained variant of Cosmos3-Nano.

Training and Evaluation Datasets

Post-Training Datasets

Data Modality: Video, Text, and LiDAR.

Post-Training Data Size:

• Text: 1 Billion to 10 Trillion Tokens
• Video: 10,000 to 1 Million Hours
• LiDAR: Less than 10,000 Hours of LiDAR range/intensity-map sequences

Data Collection Method: Hybrid: Automatic/Sensors, Synthetic, Automated.

Labeling Method: Hybrid: Manually-Labeled, Automated.

Properties: The post-training datasets consist of multi-camera driving videos, associated English captions, and LiDAR observations sourced from NVIDIA-created driving datasets.

Dataset Overview

• Total Size: 1.3B data points
• Total Number of Datasets: 393 dataset entries
• Dataset partition: Training [100%], Testing [N/A — evaluation benchmarks used separately], Validation [N/A — evaluation benchmarks used separately]
• Time period for training data collection: 2024–2026
• Time period for validation data collection: N/A (standard public benchmarks)

Raw data from internal and external sources is transformed into training-ready data through multiple stages of curation, filtering, and quality review. Data acquisition spans diverse multimodal sources — robotics, autonomous driving, industrial environments, indoor and outdoor scenes, varied lighting and weather conditions, camera viewpoints, object categories, and human activities — to broaden coverage across Physical AI operating environments. Automated filtering pipelines remove corrupted, duplicate, low-quality, and restricted content. Metadata analysis, heuristic rules, and model-assisted classifiers are applied during preprocessing to flag anomalous distributions and low-diversity subsets. Human review supplements automated filtering for selected datasets, benchmark construction, and targeted quality analysis. Datasets are balanced across modalities and task categories — visual reasoning, text-to-image, text-to-video, image-to-video, audio generation, video transfer, action-conditioned generation, and action command generation — to reduce overrepresentation of narrow domains. Synthetic and simulation-based augmentation supplements coverage of rare physical interactions and edge-case scenarios. Deduplication and provenance tracking are applied across the corpus. The resulting processed data is converted into model-ready tokenized or encoded representations through modality-specific preprocessors before training begins.

Training datasets passed through multiple layers of automated and manual safeguards designed to reduce the presence of harmful or policy-violating content across categories including weapons and weapons-related instructional content, criminal planning, child sexual abuse material (CSAM), non-consensual intimate imagery (NCII), sexual content involving minors, harassment, hate speech, profanity, threats and incitement to violence, self-harm or suicide-related content, and graphic violence. Data sources are reviewed for licensing compatibility, provenance, and alignment with internal data governance and safety policies before admission into training corpora. Automated filtering pipelines combine multiple detection strategies: hash-matching against known CSAM and NCII reference databases; classifier-based moderation models trained for explicit sexual content, hate speech, violence, weapons imagery, and other restricted categories; keyword and regex-based screening for criminal-planning, threats, and self-harm phrases in text data; metadata and provenance heuristics for source-level risk signals; and embedding-based anomaly detection to surface samples that fall outside expected distributions. Human review and targeted audits supplement automated filtering for selected datasets, benchmark construction, and safety-sensitive evaluation. For multimodal Physical AI data (robotics, autonomous driving, industrial scenes), additional filtering targets invalid action trajectories, physically implausible interactions, and unsafe control sequences. Synthetic and simulation-generated data are evaluated through internal validation before inclusion. Benchmark evaluations and red-team testing are applied post-training to surface remaining safety gaps across world generation, reasoning, audio, and action tasks. No large-scale data-filtering process can guarantee complete removal of all harmful content; residual risks may remain, particularly in rare edge cases or open-world deployment settings. Ongoing monitoring and dataset review continue post-release.

For more information about the datasets used to train this model, please see the Public Summary of Training Content.

Data Modality and Training Data Size

Data Collection Method by dataset

• Hybrid: Automatic/Sensors, Synthetic, Automated

Labeling Method by dataset

• Hybrid: Manually-Labeled, Automated

Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.

Public Datasets

Private Datasets

Synthetic Datasets

Evaluation Datasets

Data Collection Method by dataset

• Hybrid: Automatic/Sensors, Synthetic, Automated

Labeling Method by dataset

• Hybrid: Manually-Labeled, Automated

Properties: The training, testing, and evaluation datasets consist of diverse multimodal video, image, audio, action, synthetic, and sensor-conditioned data sourced from NVIDIA-owned data and publicly available, commercially permissive datasets. These datasets are curated to exclude known restricted content and to support building an Omni model that learns to generate and reason about dynamic physical environments across world reasoning and generation tasks.

Quality Benchmark(s)

We provide control inputs and text prompts in the Cosmos3 Driving Simulation Evaluation Dataset to support evaluation. Cosmos3-Nano-Transfer-Auto results use these text prompts; the Cosmos-Transfer2.5-2B Multiview baseline uses different prompts.

Multi-view RGB quality

We evaluate Cosmos3-Nano-Transfer-Auto at 720p in text-to-video (T2V) and image-to-video (I2V) settings, using a guidance scale of 3. Fréchet Inception Distance (FID) compares generated and reference frame-feature distributions; Fréchet Video Distance (FVD) compares video-feature distributions, with I3D and StyleGAN-V results reported separately. DOVER evaluates video quality.

Temporal Sampson error measures geometric agreement between matched image points across frames. Adjacent Cosine Mean (ACM) measures within-view visual consistency by averaging CLIP-feature cosine similarities between each frame after the first and both its preceding frame and the first frame, with negative similarities clamped to zero. Scores are computed per camera video, then averaged across views and samples. The score ranges from 0 to 1; higher is better. ACM uses no ground-truth video and does not measure cross-view consistency. A frozen video can score highly, so ACM alone does not establish realistic motion.

Cross-view Sampson error measures agreement across camera views, evaluated over two or five view pairs.

LiDAR generation

We evaluate Cosmos3-Nano-Transfer-Auto LiDAR generation in the T2V setting with a guidance scale of 3. Bird's-eye-view (BEV) JSD compares generated and real point distributions viewed from above using a Jensen–Shannon measure; BEV MMD² uses squared maximum mean discrepancy for a complementary distribution comparison. Fréchet Utonia distance and Utonia MMD² compare distributions of features from Utonia, a pretrained point-cloud encoder that extracts features of 3D geometry. Chamfer temporal consistency (CTC) measures geometric change between adjacent ego-motion-aligned scans; vehicle CTC applies it to tracked vehicle crops aligned to object-local coordinates. Both are reported as generated-to-real CTC ratios, where 1 means matching the real scans' average temporal distance. Object support recall measures the fraction of eligible annotated objects, counted per frame, with sufficient generated LiDAR points in their box crops. Object detection F1 is the harmonic mean of precision and recall against ground-truth annotations, matching same-class detections by planar center distance below 3 m. LidarVerse results are reported for 21 and 116 scans and use a different evaluation subset.

Inference

Acceleration Engine: PyTorch, vLLM-Omni

Test Hardware: NVIDIA Blackwell GB200, NVIDIA Hopper H100, H200, H20

Inference performance numbers below are based on vLLM-Omni runs, over 35 steps of denoising with CFG, generating 201 frames.

Empty table cells indicate that measurements are not yet reported. "-" means unsupported.

Inference Performance - per NFE

Each per-NFE time is the end-to-end time from the tables below divided by 70 NFEs (35 denoising steps with CFG).

Without LiDAR (per NFE, in seconds)

With LiDAR (per NFE, in seconds)

Inference Performance - end2end

Without LiDAR (end-to-end inference time)

With LiDAR (end-to-end inference time)

Usage Example

Quick Start

This example serves the model with vLLM-Omni and generates seven synchronized camera videos, each 201 frames at 30 FPS and 832 × 480. The inputs are the daytime sample in assets/sample_data: one world-scenario (WSM) control video and one caption per camera. It does not generate LiDAR; for joint camera and LiDAR generation, see the full notebook.

1. Start the server

On a Linux host with an NVIDIA GPU (tested on one GB200), Docker, and the NVIDIA Container Toolkit:

# One 7-camera request can take longer than the default 600 s timeouts.
docker run --runtime nvidia --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --env VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \
  --env VLLM_OMNI_DLO_DP_WAVE_TIMEOUT=1800 \
  -p 127.0.0.1:8000:8000 \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano-Transfer-Auto \
    --omni --model-class-name Cosmos3MultiviewPipeline \
    --no-guardrails \
    --host 0.0.0.0 --port 8000 --init-timeout 1800

The container publishes the server on 127.0.0.1:8000 of the host. --no-guardrails disables the Cosmos3 text and video safety checks. With two GPUs, add --cfg-parallel-size 2. The first start downloads the checkpoint. The first request also compiles the transformer, so it takes longer than later ones.

2. Generate the camera videos

pip install requests huggingface_hub

import json
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

MODEL = "nvidia/Cosmos3-Nano-Transfer-Auto"
# The sample's seven cameras, in the checkpoint's camera order.
CAMERAS = [
    "camera_front_wide_120fov",
    "camera_cross_right_120fov",
    "camera_rear_right_70fov",
    "camera_rear_tele_30fov",
    "camera_rear_left_70fov",
    "camera_cross_left_120fov",
    "camera_front_tele_30fov",
]
NUM_FRAMES = 201
# Set CAPTION_FOLDER="captions_with_timestamp" to use timestamped captions, which better match the
# model's training style and have empirically produced better inference results. Set it to
# "captions" to try the original paragraph-style captions. Both formats are supported.
CAPTION_FOLDER = "captions_with_timestamp"

# Download the sample's WSM controls, captions, and negative prompt (about 4 MB).
scene = Path(snapshot_download(MODEL, allow_patterns=[
    "assets/sample_data/daytime_scene/wsm/*.mp4",
    f"assets/sample_data/daytime_scene/{CAPTION_FOLDER}/*.json",
    "assets/sample_data/daytime_scene/captions/negative_prompt.txt",
])) / "assets/sample_data/daytime_scene"

def caption(camera):
    data = json.loads((scene / CAPTION_FOLDER / f"{camera}.json").read_text())
    return data["caption_structured"]["chunk_0_300"]["caption"]

# One negative caption, applied to every camera.
negative_prompt = (scene / "captions" / "negative_prompt.txt").read_text().strip()

# One view per camera: its caption and the index of its control among the uploads.
views = [
    {"camera_key": camera, "prompt": caption(camera), "control_reference_index": index}
    for index, camera in enumerate(CAMERAS)
]
files = [
    ("input_references", (f"{camera}.mp4", (scene / "wsm" / f"{camera}.mp4").read_bytes(), "video/mp4"))
    for camera in CAMERAS
]

response = requests.post(
    "http://127.0.0.1:8000/v1/videos/sync",
    data={
        "model": MODEL,
        "prompt": views[0]["prompt"],  # Required by the API; generation uses each view's prompt.
        "size": "832x480",
        "num_frames": NUM_FRAMES,
        "fps": 30,
        "num_inference_steps": 35,
        "guidance_scale": 3,
        "flow_shift": 10,
        "seed": 0,
        "extra_params": json.dumps({
            "wsm": {},
            "multiview": {"views": views},
            "per_view_negative_prompt": negative_prompt,
            "resolution": "480",
            "aspect_ratio": "16,9",
        }),
    },
    files=files,
    timeout=1860,
)
response.raise_for_status()
Path("transfer_auto.mp4").write_bytes(response.content)

The server trims each 300-frame control to num_frames and resizes it to 832 × 480. To vary weather, lighting, or time of day while keeping the same road and traffic, edit each view's prompt.

3. Split the output by camera

The response is one camera-major MP4: 201 frames of the first camera, then 201 of the second, and so on for all seven. To split it with FFmpeg, run:

import subprocess

for index, camera in enumerate(CAMERAS):
    start, end = index * NUM_FRAMES, (index + 1) * NUM_FRAMES
    subprocess.run([
        "ffmpeg", "-v", "error", "-y", "-i", "transfer_auto.mp4",
        "-vf", f"trim=start_frame={start}:end_frame={end},setpts=N/(30*TB)",
        "-r", "30", "-an", f"{camera}.mp4",
    ], check=True)

With the daytime sample, the output looks like this, tiled as a surround view with the front-facing cameras on top and the rear-facing cameras below.

Quick Start with SGLang

SGLang serves the same model through its OpenAI-compatible Videos API and produces the same camera-major video as the example above. Two things differ: the server reads every input from a path it can open instead of accepting uploads, so the sample is downloaded into a directory that is mounted into the container, and generation runs as an asynchronous job that the client polls. For joint camera and LiDAR generation with SGLang, see the SGLang notebook.

1. Start the server

On the same kind of host, with the published lmsysorg/sglang:dev image, which includes the Cosmos3 pipelines from sgl-project/sglang#39475:

export SAMPLE_DIR="$PWD/cosmos3-daytime-sample"  # Receives the inputs; the server reads them from this path.
export SGLANG_HF_CACHE="${SGLANG_HF_CACHE:-$HOME/.cache/sglang-huggingface}"  # The container runs as root.
mkdir -p "$SAMPLE_DIR" "$SGLANG_HF_CACHE"
chmod 777 "$SGLANG_HF_CACHE"
docker pull lmsysorg/sglang:dev
docker run --init --runtime nvidia --gpus all --ipc=host \
  -v "$SGLANG_HF_CACHE:/root/.cache/huggingface" \
  -v "$SAMPLE_DIR:$SAMPLE_DIR" \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 127.0.0.1:30000:30000 \
  lmsysorg/sglang:dev \
  sglang serve \
    --model-path nvidia/Cosmos3-Nano-Transfer-Auto \
    --host 0.0.0.0 --port 30000

The container publishes the server on 127.0.0.1:30000 of the host and mounts SAMPLE_DIR at the same path, so the paths in the request below are valid inside it. The dev tag moves with SGLang's main branch, so pull it before each start. The first start downloads the checkpoint; the HTTP port opens after the weights are loaded and the warmup request finishes. With N GPUs (2, 4, or 8), add --num-gpus N --ulysses-degree N --sp-degree N to sglang serve to split the camera-major sequence across them with Ulysses sequence parallelism; --enable-cfg-parallel additionally runs the two guidance branches on separate GPU groups and doubles the GPUs required.

2. Generate the camera videos

Run this in the same shell, so SAMPLE_DIR is set:

pip install requests huggingface_hub

import json
import os
import time
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

MODEL = "nvidia/Cosmos3-Nano-Transfer-Auto"
SERVER = "http://127.0.0.1:30000"
SAMPLE_DIR = Path(os.environ["SAMPLE_DIR"]).resolve()  # Mounted into the server container at this path.
# The sample's seven cameras, in the checkpoint's camera order.
CAMERAS = [
    "camera_front_wide_120fov",
    "camera_cross_right_120fov",
    "camera_rear_right_70fov",
    "camera_rear_tele_30fov",
    "camera_rear_left_70fov",
    "camera_cross_left_120fov",
    "camera_front_tele_30fov",
]
NUM_FRAMES = 201
# Set CAPTION_FOLDER="captions_with_timestamp" to use timestamped captions, which better match the
# model's training style and have empirically produced better inference results. Set it to
# "captions" to try the original paragraph-style captions. Both formats are supported.
CAPTION_FOLDER = "captions_with_timestamp"

# Download the sample's WSM controls, captions, and negative prompt (about 4 MB) into SAMPLE_DIR.
scene = Path(snapshot_download(MODEL, local_dir=SAMPLE_DIR, allow_patterns=[
    "assets/sample_data/daytime_scene/wsm/*.mp4",
    f"assets/sample_data/daytime_scene/{CAPTION_FOLDER}/*.json",
    "assets/sample_data/daytime_scene/captions/negative_prompt.txt",
])) / "assets/sample_data/daytime_scene"

def caption(camera):
    data = json.loads((scene / CAPTION_FOLDER / f"{camera}.json").read_text())
    return data["caption_structured"]["chunk_0_300"]["caption"]

# One negative caption, applied to every camera.
negative_prompt = (scene / "captions" / "negative_prompt.txt").read_text().strip()

# One view per camera: its caption and the server-visible path of its WSM control.
views = [
    {"camera_key": camera, "prompt": caption(camera), "control_path": str(scene / "wsm" / f"{camera}.mp4")}
    for camera in CAMERAS
]

form = {
    "model": MODEL,
    "prompt": views[0]["prompt"],  # Required by the API; generation uses each view's prompt.
    "size": "832x480",
    "num_frames": str(NUM_FRAMES),
    "fps": "30",
    "num_inference_steps": "35",
    "guidance_scale": "3",
    "flow_shift": "10",
    "seed": "0",
    "extra_params": json.dumps({
        "wsm": {},
        "multiview": {"views": views},
        "per_view_negative_prompt": negative_prompt,
        "resolution": "480",
        "aspect_ratio": "16,9",
    }),
}

# Submit the job, poll it, and download the video. Nothing is uploaded: the form carries text fields only.
response = requests.post(f"{SERVER}/v1/videos", files=[(key, (None, value)) for key, value in form.items()], timeout=60)
response.raise_for_status()
job = response.json()
deadline = time.monotonic() + 3600
while job["status"] not in ("completed", "failed", "cancelled"):
    if time.monotonic() > deadline:
        raise TimeoutError(f"Job {job['id']} did not finish in time: {job}")
    time.sleep(10)
    job = requests.get(f"{SERVER}/v1/videos/{job['id']}", timeout=60).json()
if job["status"] != "completed":
    raise RuntimeError(json.dumps(job, indent=2))
video = requests.get(f"{SERVER}/v1/videos/{job['id']}/content", timeout=600)
video.raise_for_status()
Path("transfer_auto.mp4").write_bytes(video.content)

The server trims each 300-frame control to num_frames and resizes it to 832 × 480. Split the camera-major output as in step 3 above.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

Users are responsible for model inputs and outputs. Users are responsible for ensuring safe integration of this model, including implementing guardrails as well as other safety mechanisms, prior to deployment.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy subcards. Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.