UnityVideo Wan2.2-TI2V-5B
This repository contains the five-modality, three-task UnityVideo checkpoint built on Wan2.2-TI2V-5B. Source code and usage instructions are available at JIA-Lab-research/UnityVideo.
Capabilities
- Modalities: depth, DensePose, optical flow (RAFT), segmentation, skeleton.
- Tasks: text-to-RGB+modality, video-to-modality, modality-to-video.
- Architecture: joint RGB/modality self-attention, split text cross-attention, modality identity embeddings, and separate RGB/modality output heads.
Files
| File | Size | SHA-256 | Use |
|---|---|---|---|
checkpoints/unityvideo_wan22_ti2v_5b_step15000_ema.safetensors |
10,020,954,352 bytes | 0df3909e312526c46f68097958afa055868f73354fe4276d693f7ebc398e6a39 |
Inference |
checkpoints/unityvideo_wan22_ti2v_5b_step15000.safetensors |
10,020,954,352 bytes | 4ee83a43ffdbaee90e7ff6a25fe108d8d65d255cd0803e3972bbbfb5b01db48c |
Fine-tuning |
The checkpoints contain the full two-stream DiT. Download the VAE, text encoder, tokenizer, and base configuration from Wan-AI/Wan2.2-TI2V-5B.
Quick start
git clone https://github.com/JIA-Lab-research/UnityVideo.git
cd UnityVideo
pip install -e .
unityvideo-infer \
--task video2flow \
--modality depth \
--rgb-video input.mp4 \
--output depth.mp4
The CLI downloads this checkpoint and the Wan2.2 base model on first use. See the GitHub README for all inference modes, local download commands, metadata format, and distributed training.
Training recipe
- Base: Wan2.2-TI2V-5B.
- Resolution: 256 x 256, 33 frames.
- Modalities: depth, DensePose, RAFT, segmentation, skeleton.
- Modality sampling: 0.2, 0.2, 0.2, 0.4, 0.4.
- Task sampling (
text2all,video2flow,flow2video): 0.5, 0.25, 0.25. - EMA decay: 0.9999.
- Effective released step: 15,000.
Limitations
The released model was evaluated at 256 x 256 with 33 frames. Higher resolutions, longer videos, unseen modality encodings, and safety-critical uses require independent validation. The historical A14B dual-expert experiments used an RGB+depth-only recipe and are not represented by this checkpoint.
Citation
@article{huang2025unityvideo,
title={UnityVideo: Unified Multi-Modal Multi-Task Learning for Enhancing World-Aware Video Generation},
author={Huang, Jiehui and Zhang, Yuechen and He, Xu and Gao, Yuan and Cen, Zhi and Xia, Bin and Zhou, Yan and Tao, Xin and Wan, Pengfei and Jia, Jiaya},
journal={arXiv preprint arXiv:2512.07831},
year={2025}
}