Kandinsky WM 1.0: A family of models for Physical AI

Image-to-Video generation for Physical AI: autonomous driving · robotics · general physics

🤗 Checkpoints collection · 🧩 Base model (Kandinsky 5.0) · 💻 GitHub · 📝 Habr article

This repo holds the 🤖 Robotics checkpoint of the Kandinsky WM 1.0 family — DiT robot_rl, published here in two layouts: a diffusers pipeline (model_index.json, transformer/, vae/, text_encoder*/) and the GitHub-code DiT at model/robot_rl.safetensors (same weights). The other two variants are in the Kandinsky WM 1.0 collection.

---

Contents

• Overview
• Model Zoo
• Examples — 🚗 Autonomous Driving · 🤖 Robotics · 🌍 General Physics
• Results — PAI-Bench-G · RBench · Physics-IQ Verified
• Quickstart — Path A: GitHub code · Path B: Diffusers
• Acknowledgements

---

Overview

Kandinsky WM (World Model) 1.0 is a family of Image-to-Video models that adapt Kandinsky 5.0 Video Lite — a 2B-parameter latent video diffusion model (a Kandinsky5Transformer3DModel DiT paired with a HunyuanVideo VAE and Qwen2.5-VL + CLIP text encoders, trained with flow matching) — to Physical AI: video generation that is not only visually convincing but physically plausible — consistent scene geometry, object dynamics, interactions and cause-and-effect — so the model can serve as a source of synthetic training data and as a building block for world models and simulators.

The base model was domain-adapted on large corpora of 5-second scenes (first text-to-video, then a mixed text/image-to-video regime that preserves first-frame continuation), spanning autonomous driving, robotics, and a general domain (industrial processes, physical phenomena, and human–object / human–human interaction). Each checkpoint is then reinforcement-learning post-trained with GRPO against a reward model that scores the physical plausibility of the generated scene, steering the generator toward more faithful geometry, object behaviour and interactions than domain fine-tuning alone.

Each checkpoint generates 5-second, 121-frame clips at 768×512.

Model Zoo

Three domain checkpoints, same architecture — only the DiT weights differ (the VAE / text encoders are byte-identical across all three). Each is published as a one-line diffusers pipeline; a GitHub-code (DiT-only) copy lives in the same Hub repo used purely as file storage.

Examples

Four Kandinsky WM 1.0 image-to-video generations per domain (top-ranked by an internal visual review). Each grid shows the first frame of a clip — click it to play the generated video (hosted on the static_videos dataset). The first frame and prompt for every clip also live in assets/: under assets/<domain>/, frame_N.jpg and prompt_N.txt correspond to grid cell N (left→right, top→bottom).

🚗 Autonomous Driving

First frames + prompts: assets/av/.

🤖 Robotics

First frames + prompts: assets/robotics/.

🌍 General Physics

First frames + prompts: assets/general/.

Results

Kandinsky WM 1.0 on three physical-AI video benchmarks (our row in bold). RBench and Physics-IQ were run with Qwen3-VL prompt enhancers — noted above each table, with the scripts under prompt_enhancers/. Sizes are total parameter counts where publicly disclosed (MoE models note active params; proprietary/undisclosed left as —).

PAI-Bench-G (Physical AI Bench — Generation)

Leaderboard. Column abbreviations follow the PAI-Bench-G dimensions — see the leaderboard for exact definitions. Run without prompt enhancement.

Physical-AI post-training lifts the base K5 Lite by +1.2 Overall (80.5 → 81.7) and +3.0 on the Domain axis (83.0 → 86.0) — a 2B model landing between Veo-3 and the Cosmos-Predict2.5 family.

RBench

Leaderboard, Qwen evaluator tab. RBench evaluates robot-oriented image-to-video generation across five task categories and four robot embodiments.

Prompt enhancer used: prompt_enhancers/rbench/enhance_rbench_prompts_qwen_v3.py — conservative Qwen3-VL image-grounded action canonicalization.

Physics-IQ Verified

Physics-IQ Verified

Prompt enhancer used: prompt_enhancers/physics_iq/temporal_expansion.py — Qwen3-VL temporal caption expansion.

Quickstart

Two ways to run — the checkpoints ship in both formats. The GitHub-code path is primary; the diffusers path is a one-liner that already works off the Hub.

Path A — GitHub code

The base inference code is vendored as a git submodule at kandinsky-5/ (from kandinskylab/kandinsky-5).

# 0. get this repo with the submodule
git clone --recurse-submodules https://github.com/kandinskylab/kandinsky-wm.git
cd kandinsky-wm
# (already cloned without it? run: git submodule update --init --recursive)

# 1. install the base code
pip install -r kandinsky-5/requirements.txt

# 2. pull a checkpoint (repo-as-file-storage; DiT + shared VAE/text encoders)
python examples/download_checkpoint.py --domain robotics

# 3. generate (CLI) — run from inside the submodule
cd kandinsky-5
python test.py \
  --config ./configs/k5_lite_i2v_5s_sft_sd.yaml \
  --image  ../assets/robotics/frame_1.jpg \
  --prompt "$(cat ../assets/robotics/prompt_1.txt)" \
  --video_duration 5

Python variant: examples/run_i2v_github.py.

Path B — Diffusers

pip install -U diffusers transformers accelerate imageio-ffmpeg

import torch
from diffusers import Kandinsky5I2VPipeline
from diffusers.utils import export_to_video, load_image

pipe = Kandinsky5I2VPipeline.from_pretrained(
    "kandinskylab/Kandinsky-WM-1.0-I2V-5s-RO",
    torch_dtype=torch.bfloat16,
).to("cuda")

frames = pipe(
    image=load_image("assets/robotics/frame_1.jpg"),
    prompt=open("assets/robotics/prompt_1.txt").read().strip(),
    negative_prompt=("Static, 2D cartoon, cartoon, 2d animation, paintings, images, "
                     "worst quality, low quality, ugly, deformed, walking backwards"),
    height=512, width=768, num_frames=121,
    num_inference_steps=50, guidance_scale=5.0,
).frames[0]
export_to_video(frames, "robotics_out.mp4", fps=24, quality=9)

Script: examples/run_i2v_diffusers.py.

Acknowledgements

Built on Kandinsky 5.0 (HunyuanVideo VAE, Qwen2.5-VL, CLIP). Evaluated on PAI-Bench-G, Physics-IQ, and RBench.